[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:22 dist-test-slave-hj7s krb5kdc[11772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:22 dist-test-slave-hj7s krb5kdc[11772](info): set up 2 sockets
Sep 25 02:22:22 dist-test-slave-hj7s krb5kdc[11772](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:22:23.548035 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:23 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:23.676965 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33999
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:23.760691 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:23.760936 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:23.760960 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:23.762259 11785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:23.762306 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:23.762320 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:23.762333 11785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:23.762393 11785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:23.763892 11785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:23.764146 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:23.764389 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:23.769946 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:23.770051 11785 server_base.cc:1034] running on GCE node
W20260925 02:22:23.769939 11791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:23.769939 11790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:23.770479 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:23.770804 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:23.771955 11785 hybrid_clock.cc:648] HybridClock initialized: now 1790302943771933 us; error 39 us; skew 500 ppm
Sep 25 02:22:23 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302943, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:23.774467 11785 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:23.774983 11785 webserver.cc:460] Webserver started at http://127.11.125.62:46563/ using document root <none> and password file <none>
I20260925 02:22:23.775223 11785 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:23.775302 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:23.775471 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:23.776554 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "a6bc17f282cd4a188a461aec33405e7d"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f224468790b58ff95ffeec273aabcf50"
server_key_iv: "671d29d694366269dc0cde7577d35cc0"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.777017 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "a6bc17f282cd4a188a461aec33405e7d"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f224468790b58ff95ffeec273aabcf50"
server_key_iv: "671d29d694366269dc0cde7577d35cc0"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.778402 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:23.779176 11800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:23.779372 11785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:23.779481 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "a6bc17f282cd4a188a461aec33405e7d"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f224468790b58ff95ffeec273aabcf50"
server_key_iv: "671d29d694366269dc0cde7577d35cc0"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.779579 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:23.793002 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:23.794934 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:23.795116 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:23.799505 11785 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:33999
I20260925 02:22:23.799540 11852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:33999 every 8 connection(s)
I20260925 02:22:23.799970 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:23.800855 11853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:23.802120 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 11785
I20260925 02:22:23.802206 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:23.802862 11764 external_mini_cluster.cc:1428] Setting key d80e6cadba9fa5d375d4c60d1081e57a
I20260925 02:22:23.803522 11853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d: Bootstrap starting.
I20260925 02:22:23.804248 11853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:23.804579 11853 log.cc:824] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:23.805364 11853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d: No bootstrap required, opened a new log
I20260925 02:22:23.807008 11853 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } }
I20260925 02:22:23.807176 11853 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:23.807271 11853 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bc17f282cd4a188a461aec33405e7d, State: Initialized, Role: FOLLOWER
I20260925 02:22:23.807422 11853 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } }
I20260925 02:22:23.807530 11853 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:23.807575 11853 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:23.807623 11853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:23.808422 11853 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } }
I20260925 02:22:23.808554 11853 leader_election.cc:302] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6bc17f282cd4a188a461aec33405e7d; no voters:
I20260925 02:22:23.808781 11853 leader_election.cc:288] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:23.808809 11858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:23.808943 11858 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [term 1 LEADER]: Becoming Leader. State: Replica: a6bc17f282cd4a188a461aec33405e7d, State: Running, Role: LEADER
I20260925 02:22:23.809031 11858 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } }
I20260925 02:22:23.809072 11853 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:23.809544 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bc17f282cd4a188a461aec33405e7d. Latest consensus state: current_term: 1 leader_uuid: "a6bc17f282cd4a188a461aec33405e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } } }
I20260925 02:22:23.809553 11859 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6bc17f282cd4a188a461aec33405e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc17f282cd4a188a461aec33405e7d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33999 } } }
I20260925 02:22:23.809648 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:23.809657 11859 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:23.809871 11863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:23.810433 11863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:23.812186 11863 catalog_manager.cc:1347] Generated new cluster ID: 07b9bf50cffc4608a6407940d92142e6
I20260925 02:22:23.812248 11863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:22:23 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:23.825030 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:23.803810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.11.125.62:33999)
0925 02:22:23.803998 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:23.804009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:23.804240 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:22:23.806999 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:23.807005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:23.807061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:23.807212 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:23.809026 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:23.811754 (+ 2728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:23.813558 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:23.814258 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:23.815652 (+ 1394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:23.819350 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:23.819356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:23.819356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:23.819373 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:23.821891 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:23.822538 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:23.822542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:23.822542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:23.822718 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:23.823088 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:23.823089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:23.823089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:23.823464 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:23.824466 (+ 1002us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:23.824548 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:22:23.824710 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260925 02:22:23.828344 11863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:23.829011 11863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:23.831130 11863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6bc17f282cd4a188a461aec33405e7d: Generated new TSK 0
I20260925 02:22:23.831337 11863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:23.864480 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:23.964229 11881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:23.964450 11881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:23.964476 11881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:23.965907 11881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:23.965963 11881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:23.965992 11881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:23.967689 11881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:23.967944 11881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:23.968218 11881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:23.974242 11886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:23.974289 11887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:23.974480 11881 server_base.cc:1034] running on GCE node
W20260925 02:22:23.974365 11889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:23.974778 11881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:23.975054 11881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:23.976207 11881 hybrid_clock.cc:648] HybridClock initialized: now 1790302943976182 us; error 41 us; skew 500 ppm
Sep 25 02:22:23 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302943, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:23.978922 11881 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:23.979415 11881 webserver.cc:460] Webserver started at http://127.11.125.1:33011/ using document root <none> and password file <none>
I20260925 02:22:23.979693 11881 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:23.979763 11881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:23.979919 11881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:23.981102 11881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "3317c6eee53f45eab2252cd246c300c5"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f114474940616b7c90d160d77400a9ae"
server_key_iv: "bc3164f28e771f633f9cde5cb12fc874"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.981674 11881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "3317c6eee53f45eab2252cd246c300c5"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f114474940616b7c90d160d77400a9ae"
server_key_iv: "bc3164f28e771f633f9cde5cb12fc874"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.983203 11881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 02:22:23.984201 11896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:23.984406 11881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:23.984504 11881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "3317c6eee53f45eab2252cd246c300c5"
format_stamp: "Formatted at 2026-09-25 02:22:23 on dist-test-slave-hj7s"
server_key: "f114474940616b7c90d160d77400a9ae"
server_key_iv: "bc3164f28e771f633f9cde5cb12fc874"
server_key_version: "encryptionkey@0"
I20260925 02:22:23.984648 11881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:23.997540 11881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:23.999972 11881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:24.000162 11881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:24.000419 11881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:24.000767 11881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:24.000826 11881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.000862 11881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:24.000890 11881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.007218 11881 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:40105
I20260925 02:22:24.007265 12009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:40105 every 8 connection(s)
I20260925 02:22:24.007586 11881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:24.010243 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 11881
I20260925 02:22:24.010370 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:24.010562 11764 external_mini_cluster.cc:1428] Setting key db3e6d636a4b4156bafb4afd5e2a8384
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302943, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:22:24.018560 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.008395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:51343 (local address 127.11.125.62:33999)
0925 02:22:24.008471 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:24.008473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:24.008695 (+ 222us) server_negotiation.cc:407] Connection header received
0925 02:22:24.009417 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:24.009418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:24.009429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:24.009499 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:24.010653 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.011401 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.012522 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.012802 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.012861 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.015499 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:24.015501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:24.015501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:24.015509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:24.017080 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.017518 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.017519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.017519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.017582 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.017863 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.017864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.017864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.017981 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:24.018054 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:24.018357 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:22:24.018461 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:22:24.018747 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.008374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33999 (local address 127.11.125.1:51343)
0925 02:22:24.008552 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:24.008596 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:24.009174 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:24.009560 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:24.009564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:24.010220 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:24.010538 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.010551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.011530 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.011531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.012405 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.012409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.012936 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.012938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.013099 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.013640 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:24.013651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:24.015388 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:24.017183 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.017191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.017194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.017430 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.017656 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.017657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.017657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.017780 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.018059 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:24.018067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:24.018209 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 02:22:24.018322 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:22:24.018490 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260925 02:22:24.019325 12010 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33999
I20260925 02:22:24.019436 12010 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:24.019588 12010 heartbeater.cc:511] Master 127.11.125.62:33999 requested a full tablet report, sending...
I20260925 02:22:24.020161 11817 ts_manager.cc:194] Registered new tserver with Master: 3317c6eee53f45eab2252cd246c300c5 (127.11.125.1:40105)
I20260925 02:22:24.020814 11817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:51343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:24.039599 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:24.131592 12015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:24.131806 12015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:24.131831 12015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:24.133122 12015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:24.133164 12015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:24.133191 12015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:24.135109 12015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:24.135319 12015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:24.135507 12015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:24.140844 12022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:24.140893 12021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:24.140862 12024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:24.141227 12015 server_base.cc:1034] running on GCE node
I20260925 02:22:24.141436 12015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:24.141682 12015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:24.142827 12015 hybrid_clock.cc:648] HybridClock initialized: now 1790302944142803 us; error 43 us; skew 500 ppm
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:24.145561 12015 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:24.145941 12015 webserver.cc:460] Webserver started at http://127.11.125.2:34489/ using document root <none> and password file <none>
I20260925 02:22:24.146132 12015 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:24.146171 12015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:24.146268 12015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:24.147363 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "a4ad2c6d028c43e2b3f3371ced3feb69"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "c95ad8a70bfda1e847190364cf11ecd8"
server_key_iv: "47c7a422dc72ed6ea6422df30b9ae1de"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.147673 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ad2c6d028c43e2b3f3371ced3feb69"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "c95ad8a70bfda1e847190364cf11ecd8"
server_key_iv: "47c7a422dc72ed6ea6422df30b9ae1de"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.149092 12015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:24.149852 12031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.150064 12015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:24.150141 12015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "a4ad2c6d028c43e2b3f3371ced3feb69"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "c95ad8a70bfda1e847190364cf11ecd8"
server_key_iv: "47c7a422dc72ed6ea6422df30b9ae1de"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.150241 12015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:24.158998 12015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:24.161552 12015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:24.161741 12015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:24.161986 12015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:24.162321 12015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:24.162384 12015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.162427 12015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:24.162453 12015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.171460 12015 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:42707
I20260925 02:22:24.171502 12145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:42707 every 8 connection(s)
I20260925 02:22:24.171881 12015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:24.174701 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12015
I20260925 02:22:24.174808 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:24.174973 11764 external_mini_cluster.cc:1428] Setting key e370f28d21d78bc26d33294ee53bc6f2
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:22:24.184427 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.172767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:51269 (local address 127.11.125.62:33999)
0925 02:22:24.172846 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:24.172848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:24.173111 (+ 263us) server_negotiation.cc:407] Connection header received
0925 02:22:24.173903 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:24.173904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:24.173919 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:24.173998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:24.175246 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.176024 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.177239 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.177460 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.177520 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.181133 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:24.181135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:24.181135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:24.181143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:24.182758 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.183298 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.183300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.183301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.183372 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.183668 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.183669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.183669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.183853 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:24.183936 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:24.184237 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 02:22:24.184334 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:22:24.184674 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.172751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33999 (local address 127.11.125.2:51269)
0925 02:22:24.172956 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:24.173004 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:24.173767 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:24.174133 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:24.174138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:24.174837 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:24.175109 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.175115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.176122 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.176123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.177136 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.177140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.177556 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.177558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.177721 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.179455 (+ 1734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:24.179464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:24.181022 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:24.182850 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.182859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.182861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.183163 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.183457 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.183458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.183459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.183588 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.183927 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:24.183936 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:24.184091 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 02:22:24.184201 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:22:24.184389 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260925 02:22:24.185159 12146 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33999
I20260925 02:22:24.185256 12146 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:24.185415 12146 heartbeater.cc:511] Master 127.11.125.62:33999 requested a full tablet report, sending...
I20260925 02:22:24.185811 11817 ts_manager.cc:194] Registered new tserver with Master: a4ad2c6d028c43e2b3f3371ced3feb69 (127.11.125.2:42707)
I20260925 02:22:24.186209 11817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:51269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:24.205457 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:24.294271 12151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:24.294489 12151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:24.294518 12151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:24.295780 12151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:24.295825 12151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:24.295886 12151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:24.297467 12151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:24.297724 12151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:24.297959 12151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:24.303133 12159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:24.303120 12156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:24.303165 12151 server_base.cc:1034] running on GCE node
W20260925 02:22:24.303166 12157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:24.303552 12151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:24.303746 12151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:24.304894 12151 hybrid_clock.cc:648] HybridClock initialized: now 1790302944304880 us; error 27 us; skew 500 ppm
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:24.307511 12151 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:24.307890 12151 webserver.cc:460] Webserver started at http://127.11.125.3:32961/ using document root <none> and password file <none>
I20260925 02:22:24.308143 12151 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:24.308216 12151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:24.308377 12151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:24.309592 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "683e6c8242f34f25b0bec32d803f9329"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "60a000eca383b9ff4eec546c437c9a72"
server_key_iv: "b57aff510f5948bc37d27f9d5b1c6191"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.309968 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "683e6c8242f34f25b0bec32d803f9329"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "60a000eca383b9ff4eec546c437c9a72"
server_key_iv: "b57aff510f5948bc37d27f9d5b1c6191"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.311441 12151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:24.312182 12166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.312374 12151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:24.312469 12151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "683e6c8242f34f25b0bec32d803f9329"
format_stamp: "Formatted at 2026-09-25 02:22:24 on dist-test-slave-hj7s"
server_key: "60a000eca383b9ff4eec546c437c9a72"
server_key_iv: "b57aff510f5948bc37d27f9d5b1c6191"
server_key_version: "encryptionkey@0"
I20260925 02:22:24.312552 12151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:24.352008 12151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:24.354804 12151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:24.354986 12151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:24.355288 12151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:24.355666 12151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:24.355731 12151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.355768 12151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:24.355816 12151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:24.363497 12151 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:44055
I20260925 02:22:24.363560 12280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:44055 every 8 connection(s)
I20260925 02:22:24.363911 12151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:24.371374 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12151
I20260925 02:22:24.371460 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:24.371650 11764 external_mini_cluster.cc:1428] Setting key 4a8a2ac689a993d564c67e466956b058
I20260925 02:22:24.375201 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.364710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34305 (local address 127.11.125.62:33999)
0925 02:22:24.364763 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:24.364765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:24.365109 (+ 344us) server_negotiation.cc:407] Connection header received
0925 02:22:24.365844 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:24.365845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:24.365869 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:24.365940 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:24.367160 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.367696 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.368839 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.369124 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.369207 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.371919 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:24.371921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:24.371921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:24.371929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:24.373742 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.374179 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.374180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.374180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.374243 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.374530 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.374531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.374531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.374661 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:24.374746 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:24.375059 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 02:22:24.375133 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:22:24.375515 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.364661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33999 (local address 127.11.125.3:34305)
0925 02:22:24.364885 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:24.364988 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:24.365670 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:24.366043 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:24.366048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:24.366772 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:24.367046 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.367052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.367815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.367816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.368727 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:24.368729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.369228 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.369229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:24.369408 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.369965 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:24.369974 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:24.371818 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:24.373829 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.373837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.373838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.374070 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.374320 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:24.374321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:24.374321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:24.374430 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:24.374733 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:24.374738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:24.374926 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 02:22:24.375019 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:22:24.375200 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260925 02:22:24.376077 12281 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33999
I20260925 02:22:24.376183 12281 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:24.376353 12281 heartbeater.cc:511] Master 127.11.125.62:33999 requested a full tablet report, sending...
I20260925 02:22:24.376803 11817 ts_manager.cc:194] Registered new tserver with Master: 683e6c8242f34f25b0bec32d803f9329 (127.11.125.3:44055)
I20260925 02:22:24.377219 11817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:34305
I20260925 02:22:24.384258 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:24.471813 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.461177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.11.125.1:40105)
0925 02:22:24.461658 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:24.461664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:24.461678 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:22:24.461709 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:24.461710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:24.461727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:24.461813 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:24.462795 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.463573 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.464985 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.465364 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.465433 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.468161 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:24.468175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:24.468175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:24.468184 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:24.470143 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.470529 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.470532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.470532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.470605 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.470900 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.470901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.470902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.471209 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:24.471391 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:24.471437 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 02:22:24.471641 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":139,"threads_started":1}
W20260925 02:22:24.472294 11924 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42008
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:22:24 dist-test-slave-hj7s krb5kdc[11772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:24.569669 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:24.561634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.11.125.1:40105)
0925 02:22:24.561707 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:24.561710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:24.561820 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:22:24.561852 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:24.561853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:24.561869 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:24.561925 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:24.562612 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.563457 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.564223 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:24.564626 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:24.564700 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:24.567216 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:24.567217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:24.567218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:24.567241 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:24.568558 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.568953 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.568955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.568956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.569017 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:24.569235 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:24.569235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:24.569236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:24.569339 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:24.569417 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:24.569528 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:22:24.569599 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:24.570322 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 11881
I20260925 02:22:24.575443 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12015
I20260925 02:22:24.581269 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12151
I20260925 02:22:24.586321 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 11785
2026-09-25T02:22:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2114 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:26 dist-test-slave-hj7s krb5kdc[12295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:26 dist-test-slave-hj7s krb5kdc[12295](info): set up 2 sockets
Sep 25 02:22:26 dist-test-slave-hj7s krb5kdc[12295](info): commencing operation
krb5kdc: starting...
W20260925 02:22:27.862922 11764 mini_kdc.cc:121] Time spent starting KDC: real 3.255s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:27 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:27.997459 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33447
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:33447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:28.084369 12309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:28.084582 12309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:28.084606 12309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:28.085832 12309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:28.085880 12309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:28.085894 12309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:28.085907 12309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:28.085919 12309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:28.087450 12309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:33447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:28.087787 12309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:28.088049 12309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:28.093400 12314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.093395 12315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.093456 12309 server_base.cc:1034] running on GCE node
W20260925 02:22:28.093395 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.093938 12309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:28.094223 12309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:28.095391 12309 hybrid_clock.cc:648] HybridClock initialized: now 1790302948095324 us; error 74 us; skew 500 ppm
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:28.098568 12309 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:28.099040 12309 webserver.cc:460] Webserver started at http://127.11.125.62:42843/ using document root <none> and password file <none>
I20260925 02:22:28.099263 12309 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:28.099306 12309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:28.099403 12309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:28.100489 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "8698f87554f640658537579a8a0f5bb1"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "9e6aa5a7bafdd3cf3844b68106884d61"
server_key_iv: "c03497177f20fa3668d8164b58293f2c"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.100804 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "8698f87554f640658537579a8a0f5bb1"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "9e6aa5a7bafdd3cf3844b68106884d61"
server_key_iv: "c03497177f20fa3668d8164b58293f2c"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.102334 12309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:28.103243 12324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.103513 12309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:28.103585 12309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "8698f87554f640658537579a8a0f5bb1"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "9e6aa5a7bafdd3cf3844b68106884d61"
server_key_iv: "c03497177f20fa3668d8164b58293f2c"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.103700 12309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:28.132866 12309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:28.135118 12309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:28.135311 12309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:28.139716 12309 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:33447
I20260925 02:22:28.139779 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:33447 every 8 connection(s)
I20260925 02:22:28.140143 12309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:28.141251 12378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:28.143282 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12309
I20260925 02:22:28.143380 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:28.143538 11764 external_mini_cluster.cc:1428] Setting key b4408f8d90d7f9e5126e9cab2ca2674b
I20260925 02:22:28.143910 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1: Bootstrap starting.
I20260925 02:22:28.144778 12378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:28.145124 12378 log.cc:824] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:28.146139 12378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1: No bootstrap required, opened a new log
I20260925 02:22:28.148389 12378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } }
I20260925 02:22:28.148530 12378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:28.148552 12378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8698f87554f640658537579a8a0f5bb1, State: Initialized, Role: FOLLOWER
I20260925 02:22:28.148701 12378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } }
I20260925 02:22:28.148792 12378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:28.148816 12378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:28.148952 12378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:28.149724 12378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } }
I20260925 02:22:28.149873 12378 leader_election.cc:302] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8698f87554f640658537579a8a0f5bb1; no voters:
I20260925 02:22:28.150100 12378 leader_election.cc:288] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:28.150127 12383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:28.150254 12383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 8698f87554f640658537579a8a0f5bb1, State: Running, Role: LEADER
I20260925 02:22:28.150341 12383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } }
I20260925 02:22:28.150378 12378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:28.150664 12383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8698f87554f640658537579a8a0f5bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } } }
I20260925 02:22:28.150830 12383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:28.151024 12383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8698f87554f640658537579a8a0f5bb1. Latest consensus state: current_term: 1 leader_uuid: "8698f87554f640658537579a8a0f5bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8698f87554f640658537579a8a0f5bb1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33447 } } }
I20260925 02:22:28.151129 12383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1 [sys.catalog]: This master's current role is: LEADER
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:28.151494 12390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:28.152098 12390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:28.155002 12390 catalog_manager.cc:1347] Generated new cluster ID: c196b2773597459fa7a73ddcd05b052e
I20260925 02:22:28.155114 12390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:28.156255 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.144107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.11.125.62:33447)
0925 02:22:28.144384 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.144393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.144422 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:22:28.144993 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.145000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.145053 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.145178 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.146045 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.147057 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.148240 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.148774 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.148906 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.151998 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.152002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.152003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.152016 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.154467 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.154852 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.154855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.154855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.154991 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.155269 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.155270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.155271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.155563 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.155727 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.155908 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.156046 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
I20260925 02:22:28.169710 12390 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:28.170395 12390 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:28.179193 12390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8698f87554f640658537579a8a0f5bb1: Generated new TSK 0
I20260925 02:22:28.179476 12390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:28.212586 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:28.299402 12404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:28.299619 12404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:28.299646 12404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:28.300839 12404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:28.300886 12404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:28.301016 12404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:28.302582 12404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:28.302857 12404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:28.303143 12404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:28.308871 12413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.308857 12410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.308857 12411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.309175 12404 server_base.cc:1034] running on GCE node
I20260925 02:22:28.309444 12404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:28.309744 12404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:28.310956 12404 hybrid_clock.cc:648] HybridClock initialized: now 1790302948310938 us; error 54 us; skew 500 ppm
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:28.314105 12404 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:28.314616 12404 webserver.cc:460] Webserver started at http://127.11.125.1:40541/ using document root <none> and password file <none>
I20260925 02:22:28.314877 12404 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:28.314955 12404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:28.315114 12404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:28.316226 12404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "1f4497c6bf844306b2127c902048fa80"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "b292d9c644723b37d03393ceaccb79c9"
server_key_iv: "7f7ed9ed3f4cc0b460ccc13a574d8e53"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.316627 12404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4497c6bf844306b2127c902048fa80"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "b292d9c644723b37d03393ceaccb79c9"
server_key_iv: "7f7ed9ed3f4cc0b460ccc13a574d8e53"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.318368 12404 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:22:28.319293 12420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.319684 12404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:28.319819 12404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "1f4497c6bf844306b2127c902048fa80"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "b292d9c644723b37d03393ceaccb79c9"
server_key_iv: "7f7ed9ed3f4cc0b460ccc13a574d8e53"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.319916 12404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:28.332546 12404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:28.335132 12404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:28.335322 12404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:28.335588 12404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:28.335922 12404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:28.335981 12404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.336025 12404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:28.336047 12404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.342402 12404 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:35485
I20260925 02:22:28.342449 12534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:35485 every 8 connection(s)
I20260925 02:22:28.342780 12404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:28.348444 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12404
I20260925 02:22:28.348551 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:28.348723 11764 external_mini_cluster.cc:1428] Setting key 98b8f3ec6e58111dfa19b9e486e153e3
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:28.353914 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.343625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:43641 (local address 127.11.125.62:33447)
0925 02:22:28.343697 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.343699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.343954 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:22:28.345004 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.345005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.345019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.345083 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.345739 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.346262 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.347325 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.347555 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.347649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.350441 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.350443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.350443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.350450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.352273 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.352698 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.352700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.352700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.352804 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.353116 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.353117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.353117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.353272 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.353384 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.353637 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.353774 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:22:28.353958 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.343527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33447 (local address 127.11.125.1:43641)
0925 02:22:28.343821 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.343861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.344828 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.345161 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.345165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.345438 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:28.345633 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.345639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.346370 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.346371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.347203 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.347205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.347663 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.347664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.347791 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.348365 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:28.348377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:28.350333 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:28.352368 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.352376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.352378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.352607 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.352876 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.352877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.352877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.353041 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.353346 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:28.353352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:28.353476 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.353558 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.353782 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":97,"threads_started":1}
I20260925 02:22:28.354552 12535 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33447
I20260925 02:22:28.354674 12535 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:28.354916 12535 heartbeater.cc:511] Master 127.11.125.62:33447 requested a full tablet report, sending...
I20260925 02:22:28.355598 12342 ts_manager.cc:194] Registered new tserver with Master: 1f4497c6bf844306b2127c902048fa80 (127.11.125.1:35485)
I20260925 02:22:28.356338 12342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:43641
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:28.379803 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:28.463440 12540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:28.463749 12540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:28.463793 12540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:28.465109 12540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:28.465184 12540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:28.465231 12540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:28.466859 12540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:28.467182 12540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:28.467443 12540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:28.473109 12546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.473114 12548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.473120 12545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.473217 12540 server_base.cc:1034] running on GCE node
I20260925 02:22:28.473685 12540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:28.473987 12540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:28.475167 12540 hybrid_clock.cc:648] HybridClock initialized: now 1790302948475132 us; error 46 us; skew 500 ppm
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:28.477977 12540 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:28.478471 12540 webserver.cc:460] Webserver started at http://127.11.125.2:37629/ using document root <none> and password file <none>
I20260925 02:22:28.478749 12540 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:28.478827 12540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:28.478991 12540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:28.480185 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "39b364a1896449d9ab83f44fa3104bba"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "ac376ac8c349b46ac684c9b2ce7f7e3c"
server_key_iv: "ea9d15f834096ca3b93a08c3a6ebcdba"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.480592 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "39b364a1896449d9ab83f44fa3104bba"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "ac376ac8c349b46ac684c9b2ce7f7e3c"
server_key_iv: "ea9d15f834096ca3b93a08c3a6ebcdba"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.482103 12540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:28.483016 12555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.483235 12540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:28.483361 12540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "39b364a1896449d9ab83f44fa3104bba"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "ac376ac8c349b46ac684c9b2ce7f7e3c"
server_key_iv: "ea9d15f834096ca3b93a08c3a6ebcdba"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.483444 12540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:28.503559 12540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:28.505985 12540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:28.506168 12540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:28.506436 12540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:28.506786 12540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:28.506851 12540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.506887 12540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:28.506934 12540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.512987 12540 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:34089
I20260925 02:22:28.513059 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:34089 every 8 connection(s)
I20260925 02:22:28.513429 12540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:28.515367 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12540
I20260925 02:22:28.515441 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:28.515621 11764 external_mini_cluster.cc:1428] Setting key 861d40e2e9639e40ecaee398e4555416
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:22:28.524792 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.514250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:54787 (local address 127.11.125.62:33447)
0925 02:22:28.514360 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.514362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.514578 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:22:28.515554 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.515556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.515569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.515619 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.516393 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.516967 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.518019 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.518243 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.518338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.520946 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.520948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.520948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.520958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.523067 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.523581 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.523583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.523584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.523654 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.524014 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.524015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.524015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.524188 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.524266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.524561 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.524675 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:22:28.525022 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.514190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33447 (local address 127.11.125.2:54787)
0925 02:22:28.514409 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.514448 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.515408 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.515691 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.515704 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.516074 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:28.516288 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.516293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.517082 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.517083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.517918 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.517921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.518351 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.518352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.518497 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.519080 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:28.519090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:28.520791 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:28.523213 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.523223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.523228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.523487 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.523810 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.523810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.523811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.523931 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.524271 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:28.524278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:28.524411 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.524505 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.524740 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260925 02:22:28.525684 12670 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33447
I20260925 02:22:28.525815 12670 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:28.526012 12670 heartbeater.cc:511] Master 127.11.125.62:33447 requested a full tablet report, sending...
I20260925 02:22:28.526608 12342 ts_manager.cc:194] Registered new tserver with Master: 39b364a1896449d9ab83f44fa3104bba (127.11.125.2:34089)
I20260925 02:22:28.527009 12342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:54787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:28.544656 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:28.627954 12675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:28.628226 12675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:28.628269 12675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:28.629477 12675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:28.629554 12675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:28.629601 12675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:28.631157 12675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:28.631474 12675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:28.631731 12675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:28.637049 12681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.637101 12675 server_base.cc:1034] running on GCE node
W20260925 02:22:28.637043 12680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:28.637043 12683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:28.637488 12675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:28.637768 12675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:28.638942 12675 hybrid_clock.cc:648] HybridClock initialized: now 1790302948638918 us; error 76 us; skew 500 ppm
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:28.641559 12675 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:28.642105 12675 webserver.cc:460] Webserver started at http://127.11.125.3:37977/ using document root <none> and password file <none>
I20260925 02:22:28.642359 12675 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:28.642431 12675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:28.642580 12675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:28.643653 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "d00cc89c509a4745ac52951db6285922"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "c8548d93ca31458964712fed901625ec"
server_key_iv: "02c45930906974af0bec5e85e0a7a4d3"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.644027 12675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "d00cc89c509a4745ac52951db6285922"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "c8548d93ca31458964712fed901625ec"
server_key_iv: "02c45930906974af0bec5e85e0a7a4d3"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.645489 12675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:28.646260 12690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.646430 12675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:28.646544 12675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "d00cc89c509a4745ac52951db6285922"
format_stamp: "Formatted at 2026-09-25 02:22:28 on dist-test-slave-hj7s"
server_key: "c8548d93ca31458964712fed901625ec"
server_key_iv: "02c45930906974af0bec5e85e0a7a4d3"
server_key_version: "encryptionkey@0"
I20260925 02:22:28.646623 12675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:28.658167 12675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:28.660156 12675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:28.660358 12675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:28.660609 12675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:28.661024 12675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:28.661084 12675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.661129 12675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:28.661156 12675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:28.667282 12675 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:44921
I20260925 02:22:28.667341 12805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:44921 every 8 connection(s)
I20260925 02:22:28.667704 12675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:28.669842 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12675
I20260925 02:22:28.669970 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:28.670116 11764 external_mini_cluster.cc:1428] Setting key e27ea7b9e01b6fa34e5b05c7ba3c0fc6
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:28.677827 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.668535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:50959 (local address 127.11.125.62:33447)
0925 02:22:28.668601 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.668603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.668948 (+ 345us) server_negotiation.cc:407] Connection header received
0925 02:22:28.670008 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.670009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.670024 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.670092 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.670949 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.671626 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.672660 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.672846 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.672981 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.675107 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.675109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.675109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.675116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.676409 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.676858 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.676858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.676859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.676997 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.677268 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.677269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.677269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.677384 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.677438 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.677691 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.677760 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:22:28.678107 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.668474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33447 (local address 127.11.125.3:50959)
0925 02:22:28.668704 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.668751 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.669880 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.670167 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.670177 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.670548 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:28.670838 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.670844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.671721 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.671722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.672557 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.672560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.673001 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.673002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.673114 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.673663 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:28.673671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:28.675022 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:28.676486 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.676492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.676494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.676779 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.677079 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:28.677080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:28.677080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:28.677195 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:28.677449 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:28.677453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:28.677573 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.677660 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.677827 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260925 02:22:28.678771 12806 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33447
I20260925 02:22:28.678916 12806 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:28.679133 12806 heartbeater.cc:511] Master 127.11.125.62:33447 requested a full tablet report, sending...
I20260925 02:22:28.679629 12342 ts_manager.cc:194] Registered new tserver with Master: d00cc89c509a4745ac52951db6285922 (127.11.125.3:44921)
I20260925 02:22:28.680096 12342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:50959
I20260925 02:22:28.682319 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:22:28.690797 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.684140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.11.125.62:33447)
0925 02:22:28.684198 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.684200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.684258 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:22:28.684325 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.684326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.684344 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.684418 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.685325 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.685802 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.686508 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.686758 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.686814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.688036 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.688037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.688038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.688044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.689523 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.689905 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.689906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.689907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.689977 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.690179 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.690179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.690179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.690350 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.690429 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.690651 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.690716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:22:28.693578 12342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:28.695205 12342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:28.702330 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.698165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.11.125.2:34089)
0925 02:22:28.698369 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.698375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.698385 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:28.698428 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.698430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.698444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.698519 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:28.698739 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.699583 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701115 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.701850 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701923 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.701985 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.702196 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.702238 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260925 02:22:28.702312 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.697880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35485 (local address 127.0.0.1:47076)
0925 02:22:28.698132 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.698147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.698218 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.698705 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.698707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.698709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:28.698768 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.698770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.700152 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.700153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.701241 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.701243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.701953 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.702045 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.702059 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.702128 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.702208 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":4,"spinlock_wait_cycles":8576,"thread_start_us":111,"threads_started":1}
I20260925 02:22:28.702414 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.697883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:44921 (local address 127.0.0.1:54016)
0925 02:22:28.698334 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.698343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.698382 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.698921 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.698921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.698924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:28.698970 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.698972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.700201 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.700202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.701267 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.701269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.702148 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.702148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.702214 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.702217 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.702272 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.702304 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260925 02:22:28.702631 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.697880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:34089 (local address 127.0.0.1:53090)
0925 02:22:28.698100 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.698146 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.698218 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.698568 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.698571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.698573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:28.698647 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.698650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.699693 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.699694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.701023 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.701026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.702021 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.702022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.702094 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.702099 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.702147 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.702562 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":361,"thread_start_us":49,"threads_started":1}
I20260925 02:22:28.702812 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.698147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.11.125.3:44921)
0925 02:22:28.698693 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.698698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.698708 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:28.698746 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.698748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.698762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.698834 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:28.699060 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.700114 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701361 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.702048 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.702118 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.702171 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.702310 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.702357 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":79,"threads_started":1}
I20260925 02:22:28.702864 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.698348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.11.125.1:35485)
0925 02:22:28.698523 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.698527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.698540 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:28.698581 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.698583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.698598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.698852 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:28.698935 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.700045 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701330 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.701845 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.701949 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.702011 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.702734 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.702785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260925 02:22:28.703424 12603 tablet_service.cc:1467] Processing CreateTablet for tablet 475ab583ad54458cade9e585f95003ee (DEFAULT_TABLE table=test-table [id=b9373a5ab72e4944afa80db8242555f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:28.703528 12468 tablet_service.cc:1467] Processing CreateTablet for tablet 475ab583ad54458cade9e585f95003ee (DEFAULT_TABLE table=test-table [id=b9373a5ab72e4944afa80db8242555f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:28.703644 12738 tablet_service.cc:1467] Processing CreateTablet for tablet 475ab583ad54458cade9e585f95003ee (DEFAULT_TABLE table=test-table [id=b9373a5ab72e4944afa80db8242555f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:28.703816 12603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ab583ad54458cade9e585f95003ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:28.703898 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ab583ad54458cade9e585f95003ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:28.703920 12738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475ab583ad54458cade9e585f95003ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:28.706821 12823 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Bootstrap starting.
I20260925 02:22:28.707358 12825 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Bootstrap starting.
I20260925 02:22:28.707759 12823 tablet_bootstrap.cc:654] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:28.708132 12825 tablet_bootstrap.cc:654] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:28.708669 12823 log.cc:824] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:28.708935 12825 log.cc:824] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:28.709074 12824 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: Bootstrap starting.
I20260925 02:22:28.709638 12823 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: No bootstrap required, opened a new log
I20260925 02:22:28.709710 12823 ts_tablet_manager.cc:1397] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:22:28.709725 12824 tablet_bootstrap.cc:654] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:28.710215 12824 log.cc:824] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:28.710939 12824 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: No bootstrap required, opened a new log
I20260925 02:22:28.710995 12824 ts_tablet_manager.cc:1397] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:28.711617 12823 raft_consensus.cc:348] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.711812 12823 raft_consensus.cc:374] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:28.711860 12823 raft_consensus.cc:729] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b364a1896449d9ab83f44fa3104bba, State: Initialized, Role: FOLLOWER
I20260925 02:22:28.711985 12823 consensus_queue.cc:260] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.712353 12670 heartbeater.cc:503] Master 127.11.125.62:33447 was elected leader, sending a full tablet report...
I20260925 02:22:28.712670 12823 ts_tablet_manager.cc:1428] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:22:28.713056 12824 raft_consensus.cc:348] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.713231 12824 raft_consensus.cc:374] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:28.713271 12824 raft_consensus.cc:729] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00cc89c509a4745ac52951db6285922, State: Initialized, Role: FOLLOWER
I20260925 02:22:28.713403 12824 consensus_queue.cc:260] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.713716 12824 ts_tablet_manager.cc:1428] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:28.713866 12806 heartbeater.cc:503] Master 127.11.125.62:33447 was elected leader, sending a full tablet report...
I20260925 02:22:28.713932 12825 tablet_bootstrap.cc:492] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: No bootstrap required, opened a new log
I20260925 02:22:28.714012 12825 ts_tablet_manager.cc:1397] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 02:22:28.715952 12825 raft_consensus.cc:348] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.716094 12825 raft_consensus.cc:374] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:28.716116 12825 raft_consensus.cc:729] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4497c6bf844306b2127c902048fa80, State: Initialized, Role: FOLLOWER
I20260925 02:22:28.716259 12825 consensus_queue.cc:260] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.716566 12825 ts_tablet_manager.cc:1428] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:28.716611 12535 heartbeater.cc:503] Master 127.11.125.62:33447 was elected leader, sending a full tablet report...
W20260925 02:22:28.764079 12671 tablet.cc:2369] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:28.791857 12830 raft_consensus.cc:482] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:28.791980 12830 raft_consensus.cc:504] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.792338 12830 leader_election.cc:288] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4497c6bf844306b2127c902048fa80 (127.11.125.1:35485), 39b364a1896449d9ab83f44fa3104bba (127.11.125.2:34089)
I20260925 02:22:28.795297 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.792511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35485 (local address 127.11.125.3:34103)
0925 02:22:28.792573 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.792586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.792631 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.792823 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.792824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.792826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:28.792904 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.792906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.793630 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.793631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.794430 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.794433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.795042 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.795043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.795125 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.795129 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.795183 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.795226 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:22:28.795329 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.792604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34103 (local address 127.11.125.1:35485)
0925 02:22:28.792648 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.792650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.792660 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:28.792690 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.792691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.792703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.792777 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:28.793008 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.793541 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.794537 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.794952 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.795008 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.795015 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.795210 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.795251 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:22:28.795538 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.792632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:54989 (local address 127.11.125.2:34089)
0925 02:22:28.792679 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.792681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.792854 (+ 173us) server_negotiation.cc:407] Connection header received
0925 02:22:28.792885 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.792886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.792923 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.792979 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:28.793155 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.793669 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.794666 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.795095 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.795172 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.795181 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.795416 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.795468 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:22:28.795780 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.792535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:34089 (local address 127.11.125.3:54989)
0925 02:22:28.792751 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:28.792760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:28.792810 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:28.793003 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:28.793004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:28.793005 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:28.793079 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.793081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.793760 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.793761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.794571 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:28.794574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.795210 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.795211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:28.795309 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.795313 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:28.795658 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 02:22:28.795704 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20260925 02:22:28.795815 12489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ab583ad54458cade9e585f95003ee" candidate_uuid: "d00cc89c509a4745ac52951db6285922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4497c6bf844306b2127c902048fa80" is_pre_election: true
I20260925 02:22:28.796000 12489 raft_consensus.cc:2454] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00cc89c509a4745ac52951db6285922 in term 0.
I20260925 02:22:28.796200 12624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ab583ad54458cade9e585f95003ee" candidate_uuid: "d00cc89c509a4745ac52951db6285922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b364a1896449d9ab83f44fa3104bba" is_pre_election: true
I20260925 02:22:28.796314 12694 leader_election.cc:302] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4497c6bf844306b2127c902048fa80, d00cc89c509a4745ac52951db6285922; no voters:
I20260925 02:22:28.796365 12624 raft_consensus.cc:2454] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00cc89c509a4745ac52951db6285922 in term 0.
I20260925 02:22:28.796452 12830 raft_consensus.cc:2781] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:28.796536 12830 raft_consensus.cc:482] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:28.796573 12830 raft_consensus.cc:3037] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:28.797518 12830 raft_consensus.cc:504] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.797681 12830 leader_election.cc:288] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4497c6bf844306b2127c902048fa80 (127.11.125.1:35485), 39b364a1896449d9ab83f44fa3104bba (127.11.125.2:34089)
I20260925 02:22:28.797822 12489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ab583ad54458cade9e585f95003ee" candidate_uuid: "d00cc89c509a4745ac52951db6285922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4497c6bf844306b2127c902048fa80"
I20260925 02:22:28.797883 12624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475ab583ad54458cade9e585f95003ee" candidate_uuid: "d00cc89c509a4745ac52951db6285922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b364a1896449d9ab83f44fa3104bba"
I20260925 02:22:28.797953 12489 raft_consensus.cc:3037] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:28.797955 12624 raft_consensus.cc:3037] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:28.798723 12624 raft_consensus.cc:2454] T 475ab583ad54458cade9e585f95003ee P 39b364a1896449d9ab83f44fa3104bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00cc89c509a4745ac52951db6285922 in term 1.
I20260925 02:22:28.798915 12489 raft_consensus.cc:2454] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00cc89c509a4745ac52951db6285922 in term 1.
I20260925 02:22:28.798918 12692 leader_election.cc:302] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b364a1896449d9ab83f44fa3104bba, d00cc89c509a4745ac52951db6285922; no voters:
I20260925 02:22:28.799026 12830 raft_consensus.cc:2781] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:28.799189 12830 raft_consensus.cc:686] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [term 1 LEADER]: Becoming Leader. State: Replica: d00cc89c509a4745ac52951db6285922, State: Running, Role: LEADER
I20260925 02:22:28.799433 12830 consensus_queue.cc:237] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } }
I20260925 02:22:28.800379 12341 catalog_manager.cc:5495] T 475ab583ad54458cade9e585f95003ee P d00cc89c509a4745ac52951db6285922 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00cc89c509a4745ac52951db6285922 (127.11.125.3). New cstate: current_term: 1 leader_uuid: "d00cc89c509a4745ac52951db6285922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f4497c6bf844306b2127c902048fa80" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00cc89c509a4745ac52951db6285922" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 44921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b364a1896449d9ab83f44fa3104bba" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 34089 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:28.823020 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.815035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.11.125.1:35485)
0925 02:22:28.815093 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.815095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.815213 (+ 118us) server_negotiation.cc:407] Connection header received
0925 02:22:28.815268 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.815269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.815278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.815325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.816123 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.816640 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.817440 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.817633 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.817690 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.820131 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.820144 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.820144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.820151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.821908 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.822270 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.822273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.822273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.822329 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.822492 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.822492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.822493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.822617 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.822773 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.822816 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.822885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
W20260925 02:22:28.843556 12536 tablet.cc:2369] T 475ab583ad54458cade9e585f95003ee P 1f4497c6bf844306b2127c902048fa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:28.846341 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.838885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.11.125.1:35485)
0925 02:22:28.838934 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.838935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.838999 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:22:28.839043 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.839044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.839053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.839101 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.839626 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.840090 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.841135 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.841507 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.841586 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.843920 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.843922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.843922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.843928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.845404 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.845709 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.845710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.845710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.845760 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.845928 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.845928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.845929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.846010 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.846082 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.846205 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.846280 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:28.870007 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:28.862833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.11.125.1:35485)
0925 02:22:28.862886 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:28.862888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:28.862915 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:22:28.863020 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:28.863021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:28.863038 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:28.863109 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:28.863692 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.864165 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.864993 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:28.865187 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:28.865287 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:28.867571 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:28.867572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:28.867572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:28.867579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:28.868736 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.869186 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.869187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.869187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.869263 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:28.869514 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:28.869515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:28.869515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:28.869624 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:28.869743 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:28.869825 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:22:28.869886 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 02:22:28.870530 12448 tablet_service.cc:2908] Rejecting scan request for tablet 475ab583ad54458cade9e585f95003ee: Uninitialized: clean time has not yet been initialized
I20260925 02:22:28.871399 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12404
I20260925 02:22:28.876559 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12540
I20260925 02:22:28.881997 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12675
I20260925 02:22:28.886758 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12309
2026-09-25T02:22:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4300 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12847](info): set up 2 sockets
Sep 25 02:22:28 dist-test-slave-hj7s krb5kdc[12847](info): commencing operation
krb5kdc: starting...
W20260925 02:22:29.954717 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:30.076747 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36321
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:30.160821 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:30.161134 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:30.161173 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:30.162333 12860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:30.162405 12860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:22:30.162428 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:30.162460 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:30.162487 12860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:30.162508 12860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:30.163992 12860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:36321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:30.164278 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:30.164512 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:30.169580 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.169597 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.169629 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:30.169837 12860 server_base.cc:1034] running on GCE node
I20260925 02:22:30.170174 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:30.170497 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:30.171684 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790302950171656 us; error 44 us; skew 500 ppm
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:30.174391 12860 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:30.174865 12860 webserver.cc:460] Webserver started at http://127.11.125.62:39903/ using document root <none> and password file <none>
I20260925 02:22:30.175132 12860 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:30.175207 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:30.175374 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:30.176481 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "aff545e0fec143f8a90fdae1d994c4e1"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "0c20d9b5ca378a49a2485d2276dcde94"
server_key_iv: "1891ff6b89edeb338a881673936571a5"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.176860 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "aff545e0fec143f8a90fdae1d994c4e1"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "0c20d9b5ca378a49a2485d2276dcde94"
server_key_iv: "1891ff6b89edeb338a881673936571a5"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.178306 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:30.179132 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.179319 12860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:30.179435 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "aff545e0fec143f8a90fdae1d994c4e1"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "0c20d9b5ca378a49a2485d2276dcde94"
server_key_iv: "1891ff6b89edeb338a881673936571a5"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.179514 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:30.199095 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:30.201696 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:30.201824 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:30.206579 12860 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:36321
I20260925 02:22:30.206673 12939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:36321 every 8 connection(s)
I20260925 02:22:30.207072 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:30.207863 12940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.210332 12940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Bootstrap starting.
I20260925 02:22:30.210894 12940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.211148 12940 log.cc:824] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:30.211889 12940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: No bootstrap required, opened a new log
I20260925 02:22:30.212208 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12860
I20260925 02:22:30.212318 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:30.212492 11764 external_mini_cluster.cc:1428] Setting key 260af39fe01da063886277085cf6f4be
I20260925 02:22:30.214545 12940 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } }
I20260925 02:22:30.214725 12940 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.214771 12940 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff545e0fec143f8a90fdae1d994c4e1, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.214960 12940 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } }
I20260925 02:22:30.215070 12940 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:30.215117 12940 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:30.215167 12940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.215799 12940 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } }
I20260925 02:22:30.215898 12940 leader_election.cc:302] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff545e0fec143f8a90fdae1d994c4e1; no voters:
I20260925 02:22:30.216173 12940 leader_election.cc:288] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:30.216204 12945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:30.216321 12945 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [term 1 LEADER]: Becoming Leader. State: Replica: aff545e0fec143f8a90fdae1d994c4e1, State: Running, Role: LEADER
I20260925 02:22:30.216467 12940 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:30.216468 12945 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } }
I20260925 02:22:30.216995 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff545e0fec143f8a90fdae1d994c4e1. Latest consensus state: current_term: 1 leader_uuid: "aff545e0fec143f8a90fdae1d994c4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } } }
I20260925 02:22:30.217089 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:30.217334 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff545e0fec143f8a90fdae1d994c4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff545e0fec143f8a90fdae1d994c4e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36321 } } }
I20260925 02:22:30.217463 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:30.218614 12961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:30.218713 12961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:30.218808 12952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:30.218988 12952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:30.221095 12952 catalog_manager.cc:1347] Generated new cluster ID: a69b11b09bd74cfdb6bcb8153949404c
I20260925 02:22:30.221150 12952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:30.224553 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.213062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.11.125.62:36321)
0925 02:22:30.213265 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.213276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.213311 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:22:30.214168 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.214175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.214227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.214394 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.215095 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.216203 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.217293 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.217865 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.218020 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.221182 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.221186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.221186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.221198 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.222953 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.223372 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.223375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.223376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.223466 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.223701 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.223702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.223702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.223938 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.224121 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.224217 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.224360 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260925 02:22:30.235466 12952 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:30.236014 12952 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:30.238245 12952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Generated new TSK 0
I20260925 02:22:30.238480 12952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:30.272843 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36321
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:30.373787 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:30.374013 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:30.374042 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:30.375521 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:30.375581 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:30.375604 12966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:30.375626 12966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:30.377678 12966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:30.377889 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:30.378153 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:30.383852 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.383867 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.383867 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:30.384049 12966 server_base.cc:1034] running on GCE node
I20260925 02:22:30.384389 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:30.384616 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:30.385768 12966 hybrid_clock.cc:648] HybridClock initialized: now 1790302950385754 us; error 30 us; skew 500 ppm
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:30.388386 12966 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:30.389120 12966 webserver.cc:460] Webserver started at http://127.11.125.1:36733/ using document root <none> and password file <none>
I20260925 02:22:30.389369 12966 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:30.389415 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:30.389580 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:30.390697 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "c8686f625270402fb4cd740bb1632ab7"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "50c88d337273bf914b208c3e820c877b"
server_key_iv: "446d0c22418ac40322fc556ee85d324a"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.391127 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "c8686f625270402fb4cd740bb1632ab7"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "50c88d337273bf914b208c3e820c877b"
server_key_iv: "446d0c22418ac40322fc556ee85d324a"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.392515 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:30.393534 12981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.393728 12966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:30.393837 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "c8686f625270402fb4cd740bb1632ab7"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "50c88d337273bf914b208c3e820c877b"
server_key_iv: "446d0c22418ac40322fc556ee85d324a"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.393925 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:30.404122 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:30.406229 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:30.406400 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:30.406917 12966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:30.406988 12966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.407035 12966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:30.407058 12966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:30.415462 12966 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:42039
I20260925 02:22:30.415723 13097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:42039 every 8 connection(s)
I20260925 02:22:30.416009 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:30.418488 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.407485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:59855 (local address 127.11.125.62:36321)
0925 02:22:30.407565 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.407567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.407786 (+ 219us) server_negotiation.cc:407] Connection header received
0925 02:22:30.408282 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.408284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.408298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.408375 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.409381 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.410296 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.411457 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.411900 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.412019 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.414694 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.414696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.414697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.414705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.416713 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.417296 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.417298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.417298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.417364 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.417661 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.417662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.417663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.417862 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.417942 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.418291 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.418396 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:22:30.418713 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.407402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.1:59855)
0925 02:22:30.407672 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.407702 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.408185 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.408463 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.408467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.408884 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.409232 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.409240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.410420 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.410421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.411345 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.411347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.412052 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.412054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.412203 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.412969 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.412980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.414575 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.416811 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.416817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.416819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.417165 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.417448 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.417449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.417449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.417578 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.417947 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.417955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.418131 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.418256 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.418451 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260925 02:22:30.419307 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 12966
I20260925 02:22:30.419420 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:30.419597 11764 external_mini_cluster.cc:1428] Setting key 7ae2a719585995bb610aa614a826ad51
I20260925 02:22:30.419909 13098 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36321
I20260925 02:22:30.419991 13098 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:30.420194 13098 heartbeater.cc:511] Master 127.11.125.62:36321 requested a full tablet report, sending...
I20260925 02:22:30.422173 12892 ts_manager.cc:194] Registered new tserver with Master: c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039)
I20260925 02:22:30.422995 12892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:59855
I20260925 02:22:30.427990 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.421047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.1:53413)
0925 02:22:30.421192 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.421204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.421269 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.421382 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.421383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.421620 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.421702 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.421703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.422746 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.422747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.423531 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.423534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.424034 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.424035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.424092 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.424594 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.424599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.425030 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.426838 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.426839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.426839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.427016 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.427279 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.427280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.427281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.427361 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.427703 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.427704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.427747 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.427809 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.427902 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260925 02:22:30.428056 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.421074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:53413 (local address 127.11.125.62:36321)
0925 02:22:30.421117 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.421119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.421281 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:22:30.421318 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.421318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.421328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.421369 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.421830 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.422611 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.423619 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.423936 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.424015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.425118 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.425120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.425120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.425129 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.426738 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.427099 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.427100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.427100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.427165 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.427448 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.427449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.427449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.427607 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.427688 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.427870 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.427954 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:30.448668 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36321
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:30.534497 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:30.534773 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:30.534819 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:30.536041 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:30.536110 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:30.536149 13107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:30.536187 13107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:30.537722 13107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:30.538005 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:30.538254 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:30.543478 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.543584 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.543617 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:30.543881 13107 server_base.cc:1034] running on GCE node
I20260925 02:22:30.544138 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:30.544422 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:30.545619 13107 hybrid_clock.cc:648] HybridClock initialized: now 1790302950545588 us; error 39 us; skew 500 ppm
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:30.548352 13107 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:30.548828 13107 webserver.cc:460] Webserver started at http://127.11.125.2:45839/ using document root <none> and password file <none>
I20260925 02:22:30.549098 13107 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:30.549144 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:30.549482 13107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:30.550631 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "81c42169e17645b6ab5f822d5dfceda9"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "1dce6a5c38e2b9c4bde354c6ed3df359"
server_key_iv: "4a5fe4c59a96461ac224935c54fa4f82"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.551091 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "81c42169e17645b6ab5f822d5dfceda9"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "1dce6a5c38e2b9c4bde354c6ed3df359"
server_key_iv: "4a5fe4c59a96461ac224935c54fa4f82"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.552558 13107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:30.553470 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.553684 13107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:30.553785 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "81c42169e17645b6ab5f822d5dfceda9"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "1dce6a5c38e2b9c4bde354c6ed3df359"
server_key_iv: "4a5fe4c59a96461ac224935c54fa4f82"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.553862 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:30.579034 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:30.581513 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:30.581745 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:30.582386 13107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:30.582446 13107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.582508 13107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:30.582537 13107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:30.590368 13107 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:41865
I20260925 02:22:30.590433 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:41865 every 8 connection(s)
I20260925 02:22:30.590818 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:30.592108 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.582781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:52735 (local address 127.11.125.62:36321)
0925 02:22:30.582854 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.582857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.583095 (+ 238us) server_negotiation.cc:407] Connection header received
0925 02:22:30.583641 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.583642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.583657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.583735 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.584448 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.585020 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.586056 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.586306 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.586415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.588736 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.588738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.588738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.588772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.590591 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.591067 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.591069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.591069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.591144 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.591455 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.591456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.591457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.591581 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.591652 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.591969 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.592044 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:22:30.592276 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.582790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.2:52735)
0925 02:22:30.582959 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.583001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.583542 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.583828 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.583832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.584065 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.584342 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.584350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.585113 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.585114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.585959 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.585961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.586428 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.586429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.586561 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.587095 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.587104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.588620 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.590686 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.590691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.590693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.590973 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.591234 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.591235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.591235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.591378 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.591683 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.591690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.591814 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.591913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.592042 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260925 02:22:30.592859 13239 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36321
I20260925 02:22:30.592984 13239 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:30.593176 13239 heartbeater.cc:511] Master 127.11.125.62:36321 requested a full tablet report, sending...
I20260925 02:22:30.593608 12892 ts_manager.cc:194] Registered new tserver with Master: 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865)
I20260925 02:22:30.594013 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13107
I20260925 02:22:30.594100 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:30.594141 12892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:52735
I20260925 02:22:30.594314 11764 external_mini_cluster.cc:1428] Setting key 37e4407612c893ee97c97eecc717d973
I20260925 02:22:30.600334 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.593941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.2:46865)
0925 02:22:30.594071 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.594081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.594132 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.594261 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.594262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.594480 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.594559 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.594561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.595291 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.595292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.595857 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.595864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.596331 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.596332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.596390 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.596943 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.596950 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.597394 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.599254 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.599255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.599255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.599420 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.599657 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.599657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.599658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.599713 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.600069 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.600070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.600109 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.600162 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.600259 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260925 02:22:30.600530 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.594015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:46865 (local address 127.11.125.62:36321)
0925 02:22:30.594054 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.594056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.594122 (+ 66us) server_negotiation.cc:407] Connection header received
0925 02:22:30.594180 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.594181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.594191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.594255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.594652 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.595173 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.595947 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.596254 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.596327 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.597484 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.597485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.597486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.597491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.599147 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.599503 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.599505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.599505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.599574 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.599781 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.599782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.599782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.599995 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.600074 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.600191 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.600401 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:30.623713 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36321
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:30.709362 13248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:30.709620 13248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:30.709666 13248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:30.711033 13248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:30.711118 13248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:30.711155 13248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:30.711199 13248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:30.712839 13248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:30.713168 13248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:30.713440 13248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:30.719215 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.719233 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:30.719210 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:30.719494 13248 server_base.cc:1034] running on GCE node
I20260925 02:22:30.719758 13248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:30.719990 13248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:30.721169 13248 hybrid_clock.cc:648] HybridClock initialized: now 1790302950721156 us; error 42 us; skew 500 ppm
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:30.723838 13248 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:30.724328 13248 webserver.cc:460] Webserver started at http://127.11.125.3:43797/ using document root <none> and password file <none>
I20260925 02:22:30.724593 13248 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:30.724670 13248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:30.724830 13248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:30.725988 13248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "6124c7f0e7c24f1895cb5bfe267d076e"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "4cda3861fe8f4fb7f4121cd2f5e957a0"
server_key_iv: "756f69b3fe3dc7dc320de315d0b3524d"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.726368 13248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "6124c7f0e7c24f1895cb5bfe267d076e"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "4cda3861fe8f4fb7f4121cd2f5e957a0"
server_key_iv: "756f69b3fe3dc7dc320de315d0b3524d"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.727806 13248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:30.728644 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.728832 13248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:30.728976 13248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "6124c7f0e7c24f1895cb5bfe267d076e"
format_stamp: "Formatted at 2026-09-25 02:22:30 on dist-test-slave-hj7s"
server_key: "4cda3861fe8f4fb7f4121cd2f5e957a0"
server_key_iv: "756f69b3fe3dc7dc320de315d0b3524d"
server_key_version: "encryptionkey@0"
I20260925 02:22:30.729058 13248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:30.738706 13248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:30.741266 13248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:30.741437 13248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:30.741993 13248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:30.742059 13248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.742106 13248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:30.742132 13248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:30.749719 13248 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:45887
I20260925 02:22:30.749842 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:45887 every 8 connection(s)
I20260925 02:22:30.750146 13248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:30.753214 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.742514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:33425 (local address 127.11.125.62:36321)
0925 02:22:30.742564 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.742566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.742826 (+ 260us) server_negotiation.cc:407] Connection header received
0925 02:22:30.743394 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.743395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.743409 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.743490 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.744504 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.745416 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.747169 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.747522 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.747609 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.750259 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.750261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.750261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.750270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.751656 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.752114 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.752116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.752117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.752182 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.752462 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.752463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.752463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.752601 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.752681 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.753010 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.753104 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:22:30.753408 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.742506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.3:33425)
0925 02:22:30.742691 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.742728 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.743283 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.743617 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.743623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.744035 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.744380 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.744388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.745547 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.745549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.747064 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.747068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.747622 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.747623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.747765 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.748403 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.748414 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.750143 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.751785 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.751793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.751795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.752030 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.752260 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.752261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.752262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.752384 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.752673 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.752681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.752831 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.752984 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.753161 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 02:22:30.753975 13384 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36321
I20260925 02:22:30.754089 13384 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:30.754316 13384 heartbeater.cc:511] Master 127.11.125.62:36321 requested a full tablet report, sending...
I20260925 02:22:30.754804 12892 ts_manager.cc:194] Registered new tserver with Master: 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3:45887)
I20260925 02:22:30.755336 12892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:33425
I20260925 02:22:30.759052 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13248
I20260925 02:22:30.759130 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:30.759315 11764 external_mini_cluster.cc:1428] Setting key 66f0124bd4a5659dde3836f8dfc37d8a
I20260925 02:22:30.760356 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:22:30.762650 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.755268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:38487 (local address 127.11.125.62:36321)
0925 02:22:30.755304 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.755305 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.755317 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:30.755396 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.755396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.755407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.755458 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.755932 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.756693 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.758041 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.758375 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.758454 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.759599 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.759601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.759601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.759607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.761359 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.761805 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.761807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.761807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.761864 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.762070 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.762070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.762070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.762203 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.762291 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.762466 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.762550 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:22:30.762714 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.755098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.11.125.3:38487)
0925 02:22:30.755262 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.755274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.755331 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.755560 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.755561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.755776 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.755848 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.755850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.756833 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.756834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.757963 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.757967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.758456 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.758457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.758537 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.759038 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:30.759044 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:30.759493 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:30.761449 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.761450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.761450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.761714 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.761933 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:30.761934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:30.761935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:30.762010 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:30.762284 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:30.762285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:30.762343 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.762434 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.762552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:22:30 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:30.785187 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.777557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.11.125.62:36321)
0925 02:22:30.777633 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.777635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.777646 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:22:30.777682 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.777685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.777703 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.777778 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.778412 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.778937 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.779750 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.779954 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.780082 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.782577 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:30.782578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:30.782578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:30.782587 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:30.784159 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.784493 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.784494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.784494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.784546 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:30.784708 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:30.784708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:30.784709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:30.784830 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:30.784952 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.785054 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.785118 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:22:30.787201 12892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:30.788838 12892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:30.795907 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:41865 (local address 127.0.0.1:49036)
0925 02:22:30.791699 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.791711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.791819 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.792138 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.792141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.792143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:30.792230 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.792232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793623 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.793624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.794716 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.794718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.795596 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.795597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.795659 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.795664 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.795747 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.795805 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":83,"thread_start_us":98,"threads_started":1}
I20260925 02:22:30.795917 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.11.125.2:41865)
0925 02:22:30.791736 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.791742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.791911 (+ 169us) server_negotiation.cc:407] Connection header received
0925 02:22:30.791955 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.791958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.791976 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.792049 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:30.792320 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793534 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.794802 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.795521 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.795588 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.795644 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.795772 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.795817 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260925 02:22:30.795976 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45887 (local address 127.0.0.1:49190)
0925 02:22:30.791660 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.791705 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.791821 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.792139 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.792142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.792144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:30.792259 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.792261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793638 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.793639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.794578 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.794581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.795545 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.795546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.795645 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.795662 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.795735 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.795805 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260925 02:22:30.796240 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.11.125.3:45887)
0925 02:22:30.791832 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.791836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.791848 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:30.791987 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.791990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.792007 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.792074 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:30.792340 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793550 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.794672 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.795454 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.795531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.795600 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.796043 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.796127 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":106,"threads_started":1}
I20260925 02:22:30.796746 13173 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf (DEFAULT_TABLE table=test-table [id=0604ef44cd744a0a81e3f8269665ff09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.796850 13316 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf (DEFAULT_TABLE table=test-table [id=0604ef44cd744a0a81e3f8269665ff09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.797180 13173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0fc05201d489f9caecb8d3e83fdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.797197 13316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0fc05201d489f9caecb8d3e83fdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.797585 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42039 (local address 127.0.0.1:41596)
0925 02:22:30.791945 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.791954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.792002 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.792267 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.792268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.792270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:30.792319 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.792320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793322 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.793324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.794691 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.794694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.797209 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.797211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.797329 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.797334 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.797389 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.797441 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":34,"thread_start_us":83,"threads_started":1}
I20260925 02:22:30.797891 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.791694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.11.125.1:42039)
0925 02:22:30.791896 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.791900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.792122 (+ 222us) server_negotiation.cc:407] Connection header received
0925 02:22:30.792171 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.792174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.792189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.792271 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:30.792383 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.793240 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.795677 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.797115 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.797207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.797334 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.797748 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.797789 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260925 02:22:30.800570 13407 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Bootstrap starting.
I20260925 02:22:30.800889 13408 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Bootstrap starting.
I20260925 02:22:30.801430 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf (DEFAULT_TABLE table=test-table [id=0604ef44cd744a0a81e3f8269665ff09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.801631 13407 tablet_bootstrap.cc:654] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.801631 13408 tablet_bootstrap.cc:654] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.801791 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d0fc05201d489f9caecb8d3e83fdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.802011 13407 log.cc:824] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:30.802011 13408 log.cc:824] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:30.802802 13407 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: No bootstrap required, opened a new log
I20260925 02:22:30.802877 13407 ts_tablet_manager.cc:1397] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:30.803076 13408 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: No bootstrap required, opened a new log
I20260925 02:22:30.803138 13408 ts_tablet_manager.cc:1397] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:30.804997 13411 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Bootstrap starting.
I20260925 02:22:30.805126 13408 raft_consensus.cc:348] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.805166 13407 raft_consensus.cc:348] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.805305 13407 raft_consensus.cc:374] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.805305 13408 raft_consensus.cc:374] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.805343 13407 raft_consensus.cc:729] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6124c7f0e7c24f1895cb5bfe267d076e, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.805343 13408 raft_consensus.cc:729] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c42169e17645b6ab5f822d5dfceda9, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.805465 13408 consensus_queue.cc:260] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.805506 13407 consensus_queue.cc:260] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.805670 13411 tablet_bootstrap.cc:654] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.805753 13408 ts_tablet_manager.cc:1428] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:30.805744 13407 ts_tablet_manager.cc:1428] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 02:22:30.805874 13384 heartbeater.cc:503] Master 127.11.125.62:36321 was elected leader, sending a full tablet report...
I20260925 02:22:30.805912 13239 heartbeater.cc:503] Master 127.11.125.62:36321 was elected leader, sending a full tablet report...
I20260925 02:22:30.806037 13411 log.cc:824] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:30.807056 13411 tablet_bootstrap.cc:492] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: No bootstrap required, opened a new log
I20260925 02:22:30.807113 13411 ts_tablet_manager.cc:1397] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:30.808756 13411 raft_consensus.cc:348] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.808884 13411 raft_consensus.cc:374] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.808982 13411 raft_consensus.cc:729] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8686f625270402fb4cd740bb1632ab7, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.809089 13411 consensus_queue.cc:260] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.809298 13411 ts_tablet_manager.cc:1428] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:30.809369 13098 heartbeater.cc:503] Master 127.11.125.62:36321 was elected leader, sending a full tablet report...
I20260925 02:22:30.821789 13413 raft_consensus.cc:482] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:30.821887 13413 raft_consensus.cc:504] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.822232 13413 leader_election.cc:288] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865), c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039)
I20260925 02:22:30.826094 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.822528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:36709 (local address 127.11.125.2:41865)
0925 02:22:30.822579 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.822581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.822605 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:22:30.822631 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.822632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.822643 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.822697 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:30.822924 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.823702 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825158 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.825707 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825786 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.825795 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.825988 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.826027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:22:30.826094 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.822612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:51999 (local address 127.11.125.1:42039)
0925 02:22:30.822658 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.822660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.822739 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:22:30.822766 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.822767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.822780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.822820 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:30.822972 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.823688 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825156 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.825708 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825786 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.825795 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.825988 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.826027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:22:30.826133 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.822403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:41865 (local address 127.11.125.3:36709)
0925 02:22:30.822467 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.822481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.822555 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.822707 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.822708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.822710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:30.822845 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.822847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.823776 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.823777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.825058 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.825061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.825802 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.825908 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.825910 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.825970 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.826023 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":51}
I20260925 02:22:30.826349 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.822516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42039 (local address 127.11.125.3:51999)
0925 02:22:30.822661 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.822670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.822718 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.822844 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.822844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.822846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:30.822905 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.822906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.823795 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.823796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.825058 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.825061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.825802 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.825803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.825903 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.825908 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:30.825970 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:22:30.826023 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":60,"thread_start_us":101,"threads_started":1}
I20260925 02:22:30.826462 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0fc05201d489f9caecb8d3e83fdaf" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9" is_pre_election: true
I20260925 02:22:30.826649 13193 raft_consensus.cc:2454] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 0.
I20260925 02:22:30.826638 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0fc05201d489f9caecb8d3e83fdaf" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8686f625270402fb4cd740bb1632ab7" is_pre_election: true
I20260925 02:22:30.826817 13052 raft_consensus.cc:2454] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 0.
I20260925 02:22:30.826915 13266 leader_election.cc:302] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6124c7f0e7c24f1895cb5bfe267d076e, 81c42169e17645b6ab5f822d5dfceda9; no voters:
I20260925 02:22:30.827030 13413 raft_consensus.cc:2781] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:30.827090 13413 raft_consensus.cc:482] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:30.827126 13413 raft_consensus.cc:3037] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.827950 13413 raft_consensus.cc:504] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.828090 13413 leader_election.cc:288] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 election: Requested vote from peers 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865), c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039)
I20260925 02:22:30.828265 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0fc05201d489f9caecb8d3e83fdaf" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9"
I20260925 02:22:30.828285 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d0fc05201d489f9caecb8d3e83fdaf" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8686f625270402fb4cd740bb1632ab7"
I20260925 02:22:30.828356 13052 raft_consensus.cc:3037] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.828361 13193 raft_consensus.cc:3037] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.829149 13052 raft_consensus.cc:2454] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 1.
I20260925 02:22:30.829269 13193 raft_consensus.cc:2454] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 1.
I20260925 02:22:30.829372 13266 leader_election.cc:302] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6124c7f0e7c24f1895cb5bfe267d076e, c8686f625270402fb4cd740bb1632ab7; no voters:
I20260925 02:22:30.829454 13413 raft_consensus.cc:2781] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:30.829613 13413 raft_consensus.cc:686] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 LEADER]: Becoming Leader. State: Replica: 6124c7f0e7c24f1895cb5bfe267d076e, State: Running, Role: LEADER
I20260925 02:22:30.829705 13413 consensus_queue.cc:237] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } }
I20260925 02:22:30.830458 12893 catalog_manager.cc:5495] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3). New cstate: current_term: 1 leader_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } health_report { overall_health: UNKNOWN } } }
W20260925 02:22:30.841514 13240 tablet.cc:2369] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:30.842119 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.839291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.11.125.3:45887)
0925 02:22:30.839453 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.839455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.839508 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:22:30.839600 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.839602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.839626 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.839699 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:30.839845 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.840304 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.841152 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.841345 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.841424 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:30.841657 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:30.841795 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:30.841877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:30.841944 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:22:30.841963 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 02:22:30.844131 13052 raft_consensus.cc:1261] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Refusing update from remote peer 6124c7f0e7c24f1895cb5bfe267d076e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:30.844152 13193 raft_consensus.cc:1261] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Refusing update from remote peer 6124c7f0e7c24f1895cb5bfe267d076e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:30.844534 13413 consensus_queue.cc:1035] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:30.844683 13417 consensus_queue.cc:1035] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:30.847014 13420 mvcc.cc:205] Tried to move back new op lower bound from 7333080886655238144 to 7333080886599151616. Current Snapshot: MvccSnapshot[applied={T|T < 7333080886655238144}]
I20260925 02:22:30.847448 13422 mvcc.cc:205] Tried to move back new op lower bound from 7333080886655238144 to 7333080886599151616. Current Snapshot: MvccSnapshot[applied={T|T < 7333080886655238144}]
I20260925 02:22:30.847720 13421 mvcc.cc:205] Tried to move back new op lower bound from 7333080886655238144 to 7333080886599151616. Current Snapshot: MvccSnapshot[applied={T|T < 7333080886655238144}]
I20260925 02:22:30.850916 12892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:30.851029 12892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:30.851822 12892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Sending DeleteTablet for 3 replicas of tablet 01d0fc05201d489f9caecb8d3e83fdaf
I20260925 02:22:30.852150 13173 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:30 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49036
I20260925 02:22:30.852173 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:30 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260925 02:22:30.852246 13316 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d0fc05201d489f9caecb8d3e83fdaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:30 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49190
I20260925 02:22:30.852468 13431 tablet_replica.cc:330] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: stopping tablet replica
I20260925 02:22:30.852587 13431 raft_consensus.cc:2229] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:30.852674 12892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:30.852846 12892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:30.853037 13431 raft_consensus.cc:2258] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:30.853246 13430 tablet_replica.cc:330] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: stopping tablet replica
I20260925 02:22:30.853289 13432 tablet_replica.cc:330] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: stopping tablet replica
I20260925 02:22:30.853384 13432 raft_consensus.cc:2229] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:30.853453 13430 raft_consensus.cc:2229] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:30.853509 13432 raft_consensus.cc:2258] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:30.853654 13430 raft_consensus.cc:2258] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:30.854266 13316 tablet_service.cc:1467] Processing CreateTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 (DEFAULT_TABLE table=test-table [id=22289d68f68941d18e540c27a78e5f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.854398 13316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c81ca7f52b461aa2ca2e51728a7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.854385 13032 tablet_service.cc:1467] Processing CreateTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 (DEFAULT_TABLE table=test-table [id=22289d68f68941d18e540c27a78e5f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.854387 13173 tablet_service.cc:1467] Processing CreateTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 (DEFAULT_TABLE table=test-table [id=22289d68f68941d18e540c27a78e5f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:30.854482 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c81ca7f52b461aa2ca2e51728a7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.854537 13173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c81ca7f52b461aa2ca2e51728a7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:30.856001 13411 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Bootstrap starting.
I20260925 02:22:30.856132 13408 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Bootstrap starting.
I20260925 02:22:30.856648 13411 tablet_bootstrap.cc:654] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.856859 13408 tablet_bootstrap.cc:654] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.856854 13431 ts_tablet_manager.cc:1905] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:30.857122 13407 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Bootstrap starting.
I20260925 02:22:30.857679 13408 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: No bootstrap required, opened a new log
I20260925 02:22:30.857738 13408 ts_tablet_manager.cc:1397] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:22:30.857861 13407 tablet_bootstrap.cc:654] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:30.858052 13408 raft_consensus.cc:348] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.858136 13408 raft_consensus.cc:374] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.858160 13408 raft_consensus.cc:729] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c42169e17645b6ab5f822d5dfceda9, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.858201 13408 consensus_queue.cc:260] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.858316 13408 ts_tablet_manager.cc:1428] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:30.858807 13430 ts_tablet_manager.cc:1905] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:30.859524 13431 ts_tablet_manager.cc:1918] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:30.859622 13431 log.cc:1196] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/01d0fc05201d489f9caecb8d3e83fdaf
I20260925 02:22:30.859892 13431 ts_tablet_manager.cc:1939] T 01d0fc05201d489f9caecb8d3e83fdaf P c8686f625270402fb4cd740bb1632ab7: Deleting consensus metadata
I20260925 02:22:30.860262 13432 ts_tablet_manager.cc:1905] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:30.860334 12878 catalog_manager.cc:4841] TS c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039): tablet 01d0fc05201d489f9caecb8d3e83fdaf (table test-table [id=0604ef44cd744a0a81e3f8269665ff09]) successfully deleted
I20260925 02:22:30.861354 13411 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: No bootstrap required, opened a new log
I20260925 02:22:30.861413 13411 ts_tablet_manager.cc:1397] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 02:22:30.861567 13411 raft_consensus.cc:348] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.861647 13411 raft_consensus.cc:374] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.861668 13411 raft_consensus.cc:729] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8686f625270402fb4cd740bb1632ab7, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.861711 13411 consensus_queue.cc:260] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.861801 13411 ts_tablet_manager.cc:1428] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:30.862769 13430 ts_tablet_manager.cc:1918] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:30.862833 13430 log.cc:1196] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/01d0fc05201d489f9caecb8d3e83fdaf
I20260925 02:22:30.863016 13430 ts_tablet_manager.cc:1939] T 01d0fc05201d489f9caecb8d3e83fdaf P 81c42169e17645b6ab5f822d5dfceda9: Deleting consensus metadata
I20260925 02:22:30.863477 13432 ts_tablet_manager.cc:1918] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:30.863511 12878 catalog_manager.cc:4841] TS 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865): tablet 01d0fc05201d489f9caecb8d3e83fdaf (table test-table [id=0604ef44cd744a0a81e3f8269665ff09]) successfully deleted
I20260925 02:22:30.863555 13432 log.cc:1196] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/01d0fc05201d489f9caecb8d3e83fdaf
I20260925 02:22:30.863781 13432 ts_tablet_manager.cc:1939] T 01d0fc05201d489f9caecb8d3e83fdaf P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting consensus metadata
I20260925 02:22:30.864216 13407 tablet_bootstrap.cc:492] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: No bootstrap required, opened a new log
I20260925 02:22:30.864261 13407 ts_tablet_manager.cc:1397] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260925 02:22:30.864408 13407 raft_consensus.cc:348] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.864480 13407 raft_consensus.cc:374] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:30.864499 13407 raft_consensus.cc:729] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6124c7f0e7c24f1895cb5bfe267d076e, State: Initialized, Role: FOLLOWER
I20260925 02:22:30.864563 13407 consensus_queue.cc:260] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.864667 13407 ts_tablet_manager.cc:1428] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:30.864804 12877 catalog_manager.cc:4841] TS 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3:45887): tablet 01d0fc05201d489f9caecb8d3e83fdaf (table test-table [id=0604ef44cd744a0a81e3f8269665ff09]) successfully deleted
W20260925 02:22:30.917093 13099 tablet.cc:2369] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:30.953019 13417 raft_consensus.cc:482] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:30.953114 13417 raft_consensus.cc:504] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.953313 13417 leader_election.cc:288] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039), 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865)
I20260925 02:22:30.953550 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c81ca7f52b461aa2ca2e51728a7f68" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9" is_pre_election: true
I20260925 02:22:30.953570 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c81ca7f52b461aa2ca2e51728a7f68" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8686f625270402fb4cd740bb1632ab7" is_pre_election: true
I20260925 02:22:30.953668 13193 raft_consensus.cc:2454] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 0.
I20260925 02:22:30.953682 13052 raft_consensus.cc:2454] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 0.
I20260925 02:22:30.953850 13266 leader_election.cc:302] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6124c7f0e7c24f1895cb5bfe267d076e, c8686f625270402fb4cd740bb1632ab7; no voters:
I20260925 02:22:30.953974 13417 raft_consensus.cc:2781] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:30.954025 13417 raft_consensus.cc:482] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:30.954056 13417 raft_consensus.cc:3037] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.954895 13417 raft_consensus.cc:504] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.955034 13417 leader_election.cc:288] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 election: Requested vote from peers c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039), 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865)
I20260925 02:22:30.955217 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c81ca7f52b461aa2ca2e51728a7f68" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9"
I20260925 02:22:30.955228 13052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c81ca7f52b461aa2ca2e51728a7f68" candidate_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8686f625270402fb4cd740bb1632ab7"
I20260925 02:22:30.955309 13193 raft_consensus.cc:3037] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.955327 13052 raft_consensus.cc:3037] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:30.955928 13052 raft_consensus.cc:2454] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 1.
I20260925 02:22:30.956095 13266 leader_election.cc:302] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6124c7f0e7c24f1895cb5bfe267d076e, c8686f625270402fb4cd740bb1632ab7; no voters:
I20260925 02:22:30.956143 13193 raft_consensus.cc:2454] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6124c7f0e7c24f1895cb5bfe267d076e in term 1.
I20260925 02:22:30.956207 13417 raft_consensus.cc:2781] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:30.956271 13417 raft_consensus.cc:686] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 LEADER]: Becoming Leader. State: Replica: 6124c7f0e7c24f1895cb5bfe267d076e, State: Running, Role: LEADER
I20260925 02:22:30.956328 13417 consensus_queue.cc:237] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:30.956996 12892 catalog_manager.cc:5495] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3). New cstate: current_term: 1 leader_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } health_report { overall_health: HEALTHY } } }
W20260925 02:22:31.000746 13385 tablet.cc:2369] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:22:31 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:31.009228 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.993697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.11.125.62:36321)
0925 02:22:30.993747 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:30.993749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:30.993914 (+ 165us) server_negotiation.cc:407] Connection header received
0925 02:22:30.993958 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:30.993959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:30.993969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:30.994024 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:30.994435 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.994950 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.995627 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.995825 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.995878 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.006558 (+ 10680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:31.006560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:31.006560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:31.006566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:31.007980 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.008381 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.008382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.008383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.008441 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.008645 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.008646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.008646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.008760 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:31.008834 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.008995 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.009087 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:22:31.009296 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:30.993697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36321 (local address 127.0.0.1:34924)
0925 02:22:30.993837 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:30.993846 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:30.993912 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:30.994031 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:30.994031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:30.994323 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:30.994374 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.994375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.995023 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.995024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.995556 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:30.995558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:30.995893 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:30.995893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:30.995933 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.004972 (+ 9039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:31.004977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:31.006438 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:31.008083 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:31.008093 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:31.008095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:31.008300 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:31.008507 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:31.008508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:31.008509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:31.008572 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:31.008825 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:31.008833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:31.008880 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.008970 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.009158 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":80,"thread_start_us":86,"threads_started":1}
I20260925 02:22:31.011476 12892 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:34924:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:22:31.011688 12892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:31.013264 13032 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f72971e2a436cae25b3767e945652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eae094e5b5a7438a9afbef5cf4a55919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:31.013264 13316 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f72971e2a436cae25b3767e945652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eae094e5b5a7438a9afbef5cf4a55919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:31.013306 13173 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f72971e2a436cae25b3767e945652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eae094e5b5a7438a9afbef5cf4a55919]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:31.013429 13032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f72971e2a436cae25b3767e945652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:31.013429 13316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f72971e2a436cae25b3767e945652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:31.013478 13173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f72971e2a436cae25b3767e945652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:31.014672 13408 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: Bootstrap starting.
I20260925 02:22:31.014961 13411 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: Bootstrap starting.
I20260925 02:22:31.014962 13407 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: Bootstrap starting.
I20260925 02:22:31.015166 13408 tablet_bootstrap.cc:654] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:31.015318 13407 tablet_bootstrap.cc:654] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:31.015383 13411 tablet_bootstrap.cc:654] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:31.015916 13408 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: No bootstrap required, opened a new log
I20260925 02:22:31.015969 13408 ts_tablet_manager.cc:1397] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:31.016099 13411 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: No bootstrap required, opened a new log
I20260925 02:22:31.016113 13408 raft_consensus.cc:348] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.016163 13408 raft_consensus.cc:374] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:31.016176 13408 raft_consensus.cc:729] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c42169e17645b6ab5f822d5dfceda9, State: Initialized, Role: FOLLOWER
I20260925 02:22:31.016175 13411 ts_tablet_manager.cc:1397] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:31.016234 13408 consensus_queue.cc:260] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.016327 13408 ts_tablet_manager.cc:1428] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:31.016330 13411 raft_consensus.cc:348] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.016389 13411 raft_consensus.cc:374] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:31.016412 13411 raft_consensus.cc:729] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8686f625270402fb4cd740bb1632ab7, State: Initialized, Role: FOLLOWER
I20260925 02:22:31.016390 13412 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.016459 13411 consensus_queue.cc:260] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.016510 13412 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.016585 13411 ts_tablet_manager.cc:1428] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:31.016626 13415 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.016769 13415 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.016815 13407 tablet_bootstrap.cc:492] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: No bootstrap required, opened a new log
I20260925 02:22:31.016860 13407 ts_tablet_manager.cc:1397] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:22:31.017001 13407 raft_consensus.cc:348] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.017102 13407 raft_consensus.cc:374] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:31.017125 13407 raft_consensus.cc:729] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6124c7f0e7c24f1895cb5bfe267d076e, State: Initialized, Role: FOLLOWER
I20260925 02:22:31.017181 13407 consensus_queue.cc:260] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.017287 13407 ts_tablet_manager.cc:1428] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:31.017318 13417 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.017433 13417 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.154124 13415 raft_consensus.cc:482] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:31.154232 13415 raft_consensus.cc:504] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.154585 13415 leader_election.cc:288] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865), 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3:45887)
I20260925 02:22:31.157742 13404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.154884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50317 (local address 127.11.125.2:41865)
0925 02:22:31.154928 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.154930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.154956 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:22:31.155071 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.155073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.155083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.155151 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:31.155334 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.155866 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.156937 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.157367 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.157425 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.157433 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.157616 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.157658 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:22:31.157748 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.154755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:41865 (local address 127.11.125.1:50317)
0925 02:22:31.154894 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:31.154905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:31.154976 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:31.155161 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:31.155162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:31.155164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:31.155253 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.155255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.155988 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.155989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.156825 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.156827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.157466 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.157467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.157540 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.157544 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.157599 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.157641 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260925 02:22:31.157857 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.154835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50051 (local address 127.11.125.3:45887)
0925 02:22:31.154904 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.154906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.155009 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:22:31.155113 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.155114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.155144 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.155201 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:31.155401 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.155855 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.157047 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.157468 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.157541 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.157549 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.157729 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.157777 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:22:31.157907 13447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.154786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45887 (local address 127.11.125.1:50051)
0925 02:22:31.154916 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:31.154926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:31.155010 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:31.155226 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:31.155227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:31.155229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:31.155322 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.155324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.155992 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.155993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.156949 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.156951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.157557 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.157558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.157648 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.157652 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.157714 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.157754 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":49,"thread_start_us":61,"threads_started":1}
I20260925 02:22:31.158218 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f72971e2a436cae25b3767e945652" candidate_uuid: "c8686f625270402fb4cd740bb1632ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9" is_pre_election: true
I20260925 02:22:31.158228 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f72971e2a436cae25b3767e945652" candidate_uuid: "c8686f625270402fb4cd740bb1632ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" is_pre_election: true
I20260925 02:22:31.158331 13193 raft_consensus.cc:2454] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8686f625270402fb4cd740bb1632ab7 in term 0.
I20260925 02:22:31.158360 13336 raft_consensus.cc:2454] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8686f625270402fb4cd740bb1632ab7 in term 0.
I20260925 02:22:31.158500 12983 leader_election.cc:302] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81c42169e17645b6ab5f822d5dfceda9, c8686f625270402fb4cd740bb1632ab7; no voters:
I20260925 02:22:31.158694 13415 raft_consensus.cc:2781] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:31.158737 13415 raft_consensus.cc:482] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:31.158757 13415 raft_consensus.cc:3037] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:31.159415 13415 raft_consensus.cc:504] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.159566 13415 leader_election.cc:288] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [CANDIDATE]: Term 1 election: Requested vote from peers 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865), 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3:45887)
I20260925 02:22:31.159725 13336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f72971e2a436cae25b3767e945652" candidate_uuid: "c8686f625270402fb4cd740bb1632ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6124c7f0e7c24f1895cb5bfe267d076e"
I20260925 02:22:31.159719 13193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f72971e2a436cae25b3767e945652" candidate_uuid: "c8686f625270402fb4cd740bb1632ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81c42169e17645b6ab5f822d5dfceda9"
I20260925 02:22:31.159811 13336 raft_consensus.cc:3037] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:31.159811 13193 raft_consensus.cc:3037] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:31.160542 13193 raft_consensus.cc:2454] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8686f625270402fb4cd740bb1632ab7 in term 1.
I20260925 02:22:31.160672 13336 raft_consensus.cc:2454] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8686f625270402fb4cd740bb1632ab7 in term 1.
I20260925 02:22:31.160732 12983 leader_election.cc:302] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81c42169e17645b6ab5f822d5dfceda9, c8686f625270402fb4cd740bb1632ab7; no voters:
I20260925 02:22:31.160856 13415 raft_consensus.cc:2781] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:31.160998 13415 raft_consensus.cc:686] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [term 1 LEADER]: Becoming Leader. State: Replica: c8686f625270402fb4cd740bb1632ab7, State: Running, Role: LEADER
I20260925 02:22:31.161089 13415 consensus_queue.cc:237] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } }
I20260925 02:22:31.161507 13448 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: TxnStatusTablet state changed. Reason: New leader c8686f625270402fb4cd740bb1632ab7. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.161629 13448 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:31.161808 13450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:22:31.161922 13450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:22:31.161906 12891 catalog_manager.cc:5495] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8686f625270402fb4cd740bb1632ab7 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:31.202700 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.200039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.11.125.1:42039)
0925 02:22:31.200105 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.200108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.200121 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:31.200154 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.200155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.200174 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.200289 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:31.200435 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.201037 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.201836 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.202035 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.202113 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.202319 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.202433 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.202521 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.202566 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.202593 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:22:31.202688 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.199799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42039 (local address 127.0.0.1:41604)
0925 02:22:31.199891 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:31.199903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:31.199959 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:31.200296 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:31.200297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:31.200299 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:31.200356 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.200359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.201131 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.201133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.201719 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.201721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.202112 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.202113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.202187 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.202205 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.202504 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.202516 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.202562 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.202617 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:22:31.203996 13193 raft_consensus.cc:1261] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Refusing update from remote peer c8686f625270402fb4cd740bb1632ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:31.204098 13336 raft_consensus.cc:1261] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Refusing update from remote peer c8686f625270402fb4cd740bb1632ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:31.204305 13448 consensus_queue.cc:1035] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:31.204536 13415 consensus_queue.cc:1035] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:31.205065 13412 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: TxnStatusTablet state changed. Reason: New leader c8686f625270402fb4cd740bb1632ab7. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.205169 13412 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.205338 13417 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: TxnStatusTablet state changed. Reason: New leader c8686f625270402fb4cd740bb1632ab7. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.205446 13417 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.206058 13415 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.206147 13415 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:31.206296 13412 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.206390 13412 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 81c42169e17645b6ab5f822d5dfceda9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.210160 13415 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.210294 13415 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P c8686f625270402fb4cd740bb1632ab7: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:31.210314 13417 tablet_replica.cc:439] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8686f625270402fb4cd740bb1632ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 } } peers { permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 } } peers { permanent_uuid: "6124c7f0e7c24f1895cb5bfe267d076e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45887 } } }
I20260925 02:22:31.210403 13417 tablet_replica.cc:442] T 2b1f72971e2a436cae25b3767e945652 P 6124c7f0e7c24f1895cb5bfe267d076e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:31.212924 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.210234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:42745 (local address 127.11.125.1:42039)
0925 02:22:31.210288 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.210290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.210298 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:31.210323 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.210323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.210335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.210380 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:31.210577 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.211353 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.212125 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.212436 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.212510 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.212663 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.212699 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.212752 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.212812 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.212834 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:22:31.212888 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.210116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42039 (local address 127.11.125.3:42745)
0925 02:22:31.210187 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:31.210201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:31.210249 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:31.210445 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:31.210446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:31.210448 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:31.210500 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.210502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.211443 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.211444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.212030 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.212032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.212515 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.212516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.212557 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.212577 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.212760 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.212762 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.212793 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.212833 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 02:22:31.215158 13052 raft_consensus.cc:1261] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Refusing update from remote peer 6124c7f0e7c24f1895cb5bfe267d076e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:31.215173 13193 raft_consensus.cc:1261] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Refusing update from remote peer 6124c7f0e7c24f1895cb5bfe267d076e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:31.215415 13417 consensus_queue.cc:1035] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8686f625270402fb4cd740bb1632ab7" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:31.215564 13425 consensus_queue.cc:1035] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81c42169e17645b6ab5f822d5dfceda9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:31.225260 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.222114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45887 (local address 127.11.125.1:60607)
0925 02:22:31.222316 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:31.222329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:31.222383 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:31.222506 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:31.222507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:31.222508 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:31.222552 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.222553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.223573 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.223575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.224348 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:31.224350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.224810 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.224811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:31.224849 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.224864 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.225107 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.225109 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:31.225145 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 02:22:31.225182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260925 02:22:31.225257 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.222351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:60607 (local address 127.11.125.3:45887)
0925 02:22:31.222395 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.222397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.222405 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:31.222430 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.222431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.222442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.222505 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:31.222591 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.223428 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.224432 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.224730 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.224807 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.224985 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:31.225022 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:31.225079 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.225149 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.225172 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:22:31.233459 12891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:31.233584 12891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:31.234288 12891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff545e0fec143f8a90fdae1d994c4e1: Sending DeleteTablet for 3 replicas of tablet b0c81ca7f52b461aa2ca2e51728a7f68
I20260925 02:22:31.234532 13032 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:31 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:41596
I20260925 02:22:31.234572 13173 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:31 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49036
I20260925 02:22:31.234628 13431 tablet_replica.cc:330] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: stopping tablet replica
I20260925 02:22:31.234615 13316 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c81ca7f52b461aa2ca2e51728a7f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:31 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49190
I20260925 02:22:31.234668 13430 tablet_replica.cc:330] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: stopping tablet replica
I20260925 02:22:31.234671 13431 raft_consensus.cc:2229] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:31.234694 13432 tablet_replica.cc:330] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: stopping tablet replica
I20260925 02:22:31.234710 13431 raft_consensus.cc:2258] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:31.234727 13430 raft_consensus.cc:2229] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:31.234738 13432 raft_consensus.cc:2229] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:31.234764 13430 raft_consensus.cc:2258] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:31.234790 13432 raft_consensus.cc:2258] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:31.235096 13432 ts_tablet_manager.cc:1905] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:31.235100 13431 ts_tablet_manager.cc:1905] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:31.236610 13432 ts_tablet_manager.cc:1918] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:31.236671 13432 log.cc:1196] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/b0c81ca7f52b461aa2ca2e51728a7f68
I20260925 02:22:31.236732 13431 ts_tablet_manager.cc:1918] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:31.236797 13431 log.cc:1196] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/b0c81ca7f52b461aa2ca2e51728a7f68
I20260925 02:22:31.236934 13432 ts_tablet_manager.cc:1939] T b0c81ca7f52b461aa2ca2e51728a7f68 P 6124c7f0e7c24f1895cb5bfe267d076e: Deleting consensus metadata
I20260925 02:22:31.237052 13431 ts_tablet_manager.cc:1939] T b0c81ca7f52b461aa2ca2e51728a7f68 P c8686f625270402fb4cd740bb1632ab7: Deleting consensus metadata
I20260925 02:22:31.237293 12877 catalog_manager.cc:4841] TS 6124c7f0e7c24f1895cb5bfe267d076e (127.11.125.3:45887): tablet b0c81ca7f52b461aa2ca2e51728a7f68 (table test-table [id=22289d68f68941d18e540c27a78e5f91]) successfully deleted
I20260925 02:22:31.237447 13430 ts_tablet_manager.cc:1905] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:31.239076 12878 catalog_manager.cc:4841] TS c8686f625270402fb4cd740bb1632ab7 (127.11.125.1:42039): tablet b0c81ca7f52b461aa2ca2e51728a7f68 (table test-table [id=22289d68f68941d18e540c27a78e5f91]) successfully deleted
I20260925 02:22:31.239077 13430 ts_tablet_manager.cc:1918] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:31.239140 13430 log.cc:1196] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/b0c81ca7f52b461aa2ca2e51728a7f68
I20260925 02:22:31.239349 13430 ts_tablet_manager.cc:1939] T b0c81ca7f52b461aa2ca2e51728a7f68 P 81c42169e17645b6ab5f822d5dfceda9: Deleting consensus metadata
I20260925 02:22:31.239670 12878 catalog_manager.cc:4841] TS 81c42169e17645b6ab5f822d5dfceda9 (127.11.125.2:41865): tablet b0c81ca7f52b461aa2ca2e51728a7f68 (table test-table [id=22289d68f68941d18e540c27a78e5f91]) successfully deleted
Sep 25 02:22:31 dist-test-slave-hj7s krb5kdc[12847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302950, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:31.248237 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.239324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.11.125.1:42039)
0925 02:22:31.239392 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.239394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.239596 (+ 202us) server_negotiation.cc:407] Connection header received
0925 02:22:31.239769 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.239770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.239781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.239837 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:31.240410 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.240966 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.241657 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.241870 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.241958 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.245318 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:31.245326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:31.245326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:31.245333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:31.246970 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.247406 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.247409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.247409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.247478 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.247720 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.247720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.247721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.247864 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:31.247967 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.248110 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.248172 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:22:31.248605 13092 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41608
I20260925 02:22:31.256130 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:31.249634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.11.125.62:36321)
0925 02:22:31.249681 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:31.249683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:31.249884 (+ 201us) server_negotiation.cc:407] Connection header received
0925 02:22:31.249957 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:31.249958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:31.249970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:31.250016 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:31.250525 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.251033 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.251803 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:31.252015 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:31.252101 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:31.253405 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:31.253406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:31.253406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:31.253415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:31.254919 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.255282 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.255284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.255284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.255344 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:31.255552 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:31.255553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:31.255554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:31.255666 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:31.255766 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:31.255929 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 02:22:31.256013 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:22:31.256390 12891 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34938
I20260925 02:22:31.257208 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12966
I20260925 02:22:31.264127 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13107
I20260925 02:22:31.269109 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13248
I20260925 02:22:31.275138 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 12860
2026-09-25T02:22:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2389 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:31 dist-test-slave-hj7s krb5kdc[13471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:31 dist-test-slave-hj7s krb5kdc[13471](info): set up 2 sockets
Sep 25 02:22:31 dist-test-slave-hj7s krb5kdc[13471](info): commencing operation
krb5kdc: starting...
W20260925 02:22:32.319013 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:32 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:32.440155 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:46739
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:46739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:32.524763 13484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:32.525075 13484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:32.525120 13484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:32.527761 13484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:32.527839 13484 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:22:32.527865 13484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:32.527891 13484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:32.527923 13484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:32.527954 13484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:32.530575 13484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:46739
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:32.530813 13484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:32.531016 13484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:32.536173 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:32.536275 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:32.536240 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:32.537217 13484 server_base.cc:1034] running on GCE node
I20260925 02:22:32.537525 13484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:32.537830 13484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:32.539081 13484 hybrid_clock.cc:648] HybridClock initialized: now 1790302952539060 us; error 52 us; skew 500 ppm
Sep 25 02:22:32 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:32.541947 13484 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:32.542447 13484 webserver.cc:460] Webserver started at http://127.11.125.62:38301/ using document root <none> and password file <none>
I20260925 02:22:32.542673 13484 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:32.542719 13484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:32.542815 13484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:32.543922 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "270136b009b84a0aa72de5e0fa4869b7"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "99dc21b5166ed50d4406a10c36ab49fa"
server_key_iv: "ecee3e774df4f5e2fe497979ef0e4944"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.544245 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "270136b009b84a0aa72de5e0fa4869b7"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "99dc21b5166ed50d4406a10c36ab49fa"
server_key_iv: "ecee3e774df4f5e2fe497979ef0e4944"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.545791 13484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:32.546564 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:32.546774 13484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:32.546887 13484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "270136b009b84a0aa72de5e0fa4869b7"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "99dc21b5166ed50d4406a10c36ab49fa"
server_key_iv: "ecee3e774df4f5e2fe497979ef0e4944"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.546985 13484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:32.559370 13484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:32.561947 13484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:32.562068 13484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:32.567646 13484 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:46739
I20260925 02:22:32.567754 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:46739 every 8 connection(s)
I20260925 02:22:32.568109 13484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:32.568991 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:32.571620 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Bootstrap starting.
I20260925 02:22:32.572264 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:32.572576 13562 log.cc:824] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:32.573380 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: No bootstrap required, opened a new log
I20260925 02:22:32.575196 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13484
I20260925 02:22:32.575310 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:32.575489 11764 external_mini_cluster.cc:1428] Setting key b3f60b9f3c44ff276e2c8b261c8163d0
I20260925 02:22:32.575690 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } }
I20260925 02:22:32.575888 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:32.575934 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270136b009b84a0aa72de5e0fa4869b7, State: Initialized, Role: FOLLOWER
I20260925 02:22:32.576177 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } }
I20260925 02:22:32.576290 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:32.576318 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:32.576390 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:22:32 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:32.585484 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:32.576048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.11.125.62:46739)
0925 02:22:32.576247 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:32.576255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:32.576303 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:22:32.576473 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:32.576480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:32.576538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:32.576651 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:32.577441 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.578343 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.579052 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.579454 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.579536 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:32.582212 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:32.582216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:32.582216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:32.582228 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:32.583803 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.584201 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.584204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.584205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.584317 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.584502 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.584503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.584503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.584746 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:32.584858 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:32.585110 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:22:32.585310 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260925 02:22:32.645571 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } }
I20260925 02:22:32.645823 13562 leader_election.cc:302] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 270136b009b84a0aa72de5e0fa4869b7; no voters:
I20260925 02:22:32.646114 13562 leader_election.cc:288] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:32.646204 13567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:32.646456 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:32.646476 13567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [term 1 LEADER]: Becoming Leader. State: Replica: 270136b009b84a0aa72de5e0fa4869b7, State: Running, Role: LEADER
I20260925 02:22:32.646664 13567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } }
I20260925 02:22:32.647110 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 270136b009b84a0aa72de5e0fa4869b7. Latest consensus state: current_term: 1 leader_uuid: "270136b009b84a0aa72de5e0fa4869b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } } }
I20260925 02:22:32.647327 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:32.647559 13568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "270136b009b84a0aa72de5e0fa4869b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "270136b009b84a0aa72de5e0fa4869b7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46739 } } }
I20260925 02:22:32.647925 13568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:32.648507 13583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:32.648594 13583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:32.648653 13576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:32.648836 13576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:32.650717 13576 catalog_manager.cc:1347] Generated new cluster ID: e95a709fa12f4237881708c1687c4ef8
I20260925 02:22:32.650771 13576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:32.680163 13576 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:32.681182 13576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:32.686164 13576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Generated new TSK 0
I20260925 02:22:32.686409 13576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
W20260925 02:22:32.831449 11764 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.125.1: real 0.138s user 0.000s sys 0.004s
I20260925 02:22:32.831818 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46739
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:32.915222 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:32.915416 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:32.915442 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:32.916638 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:32.916683 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:32.916704 13588 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:22:32.916749 13588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:32.916790 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:32.918326 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:32.918571 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:32.918819 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:32.924162 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:32.924271 13588 server_base.cc:1034] running on GCE node
W20260925 02:22:32.924189 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:32.924178 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:32.924610 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:32.924821 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:32.925971 13588 hybrid_clock.cc:648] HybridClock initialized: now 1790302952925955 us; error 30 us; skew 500 ppm
Sep 25 02:22:32 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:32.928490 13588 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:32.928897 13588 webserver.cc:460] Webserver started at http://127.11.125.1:38239/ using document root <none> and password file <none>
I20260925 02:22:32.929174 13588 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:32.929215 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:32.929315 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:32.930328 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "a4b6d136b239470eb93776872a9a472c"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "39ed668d7b159fa5de2ab9eda8ce6dd3"
server_key_iv: "961a5342dcdccf72bbc687a6260b6cf4"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.930644 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "a4b6d136b239470eb93776872a9a472c"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "39ed668d7b159fa5de2ab9eda8ce6dd3"
server_key_iv: "961a5342dcdccf72bbc687a6260b6cf4"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.931938 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:32.932651 13604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:32.932816 13588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:32.932897 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "a4b6d136b239470eb93776872a9a472c"
format_stamp: "Formatted at 2026-09-25 02:22:32 on dist-test-slave-hj7s"
server_key: "39ed668d7b159fa5de2ab9eda8ce6dd3"
server_key_iv: "961a5342dcdccf72bbc687a6260b6cf4"
server_key_version: "encryptionkey@0"
I20260925 02:22:32.933013 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:32.945955 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:32.948611 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:32.948738 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:32.949337 13588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:32.949383 13588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:32.949422 13588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:32.949476 13588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:32.956749 13588 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:35471
I20260925 02:22:32.956846 13720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:35471 every 8 connection(s)
I20260925 02:22:32.957218 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:22:32 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:32.957579 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13588
I20260925 02:22:32.957680 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:32.957894 11764 external_mini_cluster.cc:1428] Setting key 13c74ca7513fb58ff40093c782e447f9
I20260925 02:22:32.961561 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:32.949986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:43127 (local address 127.11.125.62:46739)
0925 02:22:32.950061 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:32.950063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:32.950344 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:22:32.951119 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:32.951121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:32.951138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:32.951208 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:32.952144 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.953055 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.954664 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.955000 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.955090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:32.957994 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:32.957996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:32.957997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:32.958007 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:32.959830 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.960379 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.960380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.960380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.960435 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.960823 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.960824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.960824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.960999 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:32.961065 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:32.961421 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:22:32.961494 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:22:32.961884 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:32.949995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.1:43127)
0925 02:22:32.950170 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:32.950209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:32.950959 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:32.951318 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:32.951322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:32.951701 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:32.952010 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:32.952018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.953181 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.953182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:32.954519 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:32.954522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.955102 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.955103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:32.955216 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:32.955882 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:32.955893 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:32.957888 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:32.959922 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:32.959929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:32.959931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:32.960291 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:32.960507 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:32.960508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:32.960508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:32.960749 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:32.961075 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:32.961083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:32.961262 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 02:22:32.961397 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 02:22:32.961601 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 02:22:32.962559 13721 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46739
I20260925 02:22:32.962670 13721 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:32.962863 13721 heartbeater.cc:511] Master 127.11.125.62:46739 requested a full tablet report, sending...
I20260925 02:22:32.963498 13515 ts_manager.cc:194] Registered new tserver with Master: a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471)
I20260925 02:22:32.964308 13515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:43127
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:22:32.969863 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:32.963673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:34555 (local address 127.11.125.62:46739)
0925 02:22:32.963714 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:32.963716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:32.963770 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:22:32.963817 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:32.963818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:32.963827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:32.963883 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:32.964283 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.964825 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.965979 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.966160 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.966218 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:32.967250 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:32.967251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:32.967251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:32.967258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:32.968700 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.969162 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.969163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.969163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.969248 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:32.969457 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:32.969457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:32.969458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:32.969562 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:32.969617 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:32.969726 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:22:32.969799 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:32.969880 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:32.963537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.1:34555)
0925 02:22:32.963680 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:32.963693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:32.963768 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:32.963901 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:32.963902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:32.964114 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:32.964198 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:32.964200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.964951 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.964952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:32.965888 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:32.965891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:32.966243 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:32.966244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:32.966331 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:32.966766 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:32.966773 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:32.967156 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:32.968836 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:32.968837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:32.968837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:32.969093 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:32.969338 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:32.969339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:32.969339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:32.969395 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:32.969620 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:32.969621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:32.969659 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:22:32.969709 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:22:32.969807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:32.988442 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46739
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:33.071805 13730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:33.072018 13730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:33.072044 13730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:33.073318 13730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:33.073362 13730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:33.073383 13730 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:22:33.073395 13730 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:33.073412 13730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:33.074890 13730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:33.075085 13730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:33.075273 13730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:33.080437 13739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:33.080451 13737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:33.080456 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:33.080703 13730 server_base.cc:1034] running on GCE node
I20260925 02:22:33.081000 13730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:33.081276 13730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:33.082491 13730 hybrid_clock.cc:648] HybridClock initialized: now 1790302953082465 us; error 34 us; skew 500 ppm
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:33.085371 13730 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:33.085852 13730 webserver.cc:460] Webserver started at http://127.11.125.2:35085/ using document root <none> and password file <none>
I20260925 02:22:33.086148 13730 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:33.086196 13730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:33.086351 13730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:33.087443 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "02dad6b1d55f410280ba8174338a2c94"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "4b1cf65b17e8d8475b6d37db41902e93"
server_key_iv: "3b40e6a824f24c2b4a28ccebb0d88f07"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.087829 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "02dad6b1d55f410280ba8174338a2c94"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "4b1cf65b17e8d8475b6d37db41902e93"
server_key_iv: "3b40e6a824f24c2b4a28ccebb0d88f07"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.089275 13730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:33.090056 13746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.090219 13730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:33.090318 13730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "02dad6b1d55f410280ba8174338a2c94"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "4b1cf65b17e8d8475b6d37db41902e93"
server_key_iv: "3b40e6a824f24c2b4a28ccebb0d88f07"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.090384 13730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:33.105396 13730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:33.107295 13730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:33.107406 13730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:33.107906 13730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:33.107947 13730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.107971 13730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:33.107985 13730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.115646 13730 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:38869
I20260925 02:22:33.115742 13862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:38869 every 8 connection(s)
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:33.116133 13730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:33.119160 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.108471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:41713 (local address 127.11.125.62:46739)
0925 02:22:33.108550 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.108552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.108778 (+ 226us) server_negotiation.cc:407] Connection header received
0925 02:22:33.109416 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.109417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.109433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.109518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.110500 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.111344 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.113198 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.113536 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.113620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.116325 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.116328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.116328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.116339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.117897 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.118311 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.118311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.118312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.118368 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.118596 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.118596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.118597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.118710 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.118765 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.119019 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.119091 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:22:33.119344 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.108456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.2:41713)
0925 02:22:33.108643 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.108685 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.109282 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.109618 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.109624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.110068 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:33.110399 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.110405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.111483 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.111484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.113044 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.113048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.113631 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.113633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.113814 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.114526 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:33.114535 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:33.116197 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:33.117988 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.117996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.117998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.118236 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.118431 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.118432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.118432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.118539 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.118772 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:33.118777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:33.118903 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.118989 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.119123 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260925 02:22:33.119865 13863 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46739
I20260925 02:22:33.119958 13863 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:33.120198 13863 heartbeater.cc:511] Master 127.11.125.62:46739 requested a full tablet report, sending...
I20260925 02:22:33.120671 13516 ts_manager.cc:194] Registered new tserver with Master: 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869)
I20260925 02:22:33.121258 13516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:41713
I20260925 02:22:33.123705 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13730
I20260925 02:22:33.123804 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:33.123997 11764 external_mini_cluster.cc:1428] Setting key 6136dc713dc2f26d71471df16bba04b9
I20260925 02:22:33.126989 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.121125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.2:38821)
0925 02:22:33.121269 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.121280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.121340 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.121531 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.121532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.121753 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:33.121818 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.121820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.122461 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.122462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.122955 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.122956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.123292 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.123293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.123349 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.123760 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:33.123766 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:33.124169 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:33.125863 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.125864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.125864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.126111 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.126348 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.126349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.126349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.126405 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.126741 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:33.126742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:33.126789 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.126840 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.126919 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260925 02:22:33.127084 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.121177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:38821 (local address 127.11.125.62:46739)
0925 02:22:33.121217 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.121219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.121339 (+ 120us) server_negotiation.cc:407] Connection header received
0925 02:22:33.121429 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.121430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.121442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.121504 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.121899 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.122366 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.123028 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.123207 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.123277 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.124265 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.124266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.124266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.124272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.125797 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.126193 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.126195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.126195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.126276 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.126469 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.126470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.126470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.126676 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.126748 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.126881 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.126974 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:33.152482 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46739
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:33.238288 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:33.238514 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:33.238543 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:33.239717 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:33.239764 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:33.239786 13872 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:22:33.239799 13872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:33.239822 13872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:33.241385 13872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:33.241595 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:33.241794 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:33.247378 13878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:33.247383 13877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:33.247457 13872 server_base.cc:1034] running on GCE node
W20260925 02:22:33.247383 13880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:33.247896 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:33.248178 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:33.249351 13872 hybrid_clock.cc:648] HybridClock initialized: now 1790302953249332 us; error 35 us; skew 500 ppm
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:33.252015 13872 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:33.252512 13872 webserver.cc:460] Webserver started at http://127.11.125.3:40475/ using document root <none> and password file <none>
I20260925 02:22:33.252713 13872 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:33.252763 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:33.252858 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:33.254012 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "009ea5b089a3465cbcd0405f182a8a7e"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "df65baf78e519948ce2aabddd545df7a"
server_key_iv: "68b764ae63fee9fbfe7e68c6aff8c958"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.254333 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "009ea5b089a3465cbcd0405f182a8a7e"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "df65baf78e519948ce2aabddd545df7a"
server_key_iv: "68b764ae63fee9fbfe7e68c6aff8c958"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.255659 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:33.256453 13887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.256654 13872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:33.256771 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "009ea5b089a3465cbcd0405f182a8a7e"
format_stamp: "Formatted at 2026-09-25 02:22:33 on dist-test-slave-hj7s"
server_key: "df65baf78e519948ce2aabddd545df7a"
server_key_iv: "68b764ae63fee9fbfe7e68c6aff8c958"
server_key_version: "encryptionkey@0"
I20260925 02:22:33.256837 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:33.279709 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:33.281687 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:33.281824 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:33.282334 13872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:33.282371 13872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.282399 13872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:33.282459 13872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.289561 13872 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:46123
I20260925 02:22:33.289650 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:46123 every 8 connection(s)
I20260925 02:22:33.289963 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:33.293081 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.282909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:55023 (local address 127.11.125.62:46739)
0925 02:22:33.282994 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.282997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.283202 (+ 205us) server_negotiation.cc:407] Connection header received
0925 02:22:33.283743 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.283744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.283760 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.283841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.284765 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.285681 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.287362 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.287710 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.287796 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.290563 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.290565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.290565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.290572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.291791 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.292176 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.292176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.292177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.292229 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.292466 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.292467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.292467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.292572 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.292621 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.292863 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.292987 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:22:33.293248 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.282852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.3:55023)
0925 02:22:33.283060 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.283104 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.283628 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.283935 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.283939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.284320 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:33.284643 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.284651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.285831 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.285833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.287258 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.287262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.287797 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.287798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.287909 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.288527 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:33.288540 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:33.290474 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:33.291867 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.291873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.291875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.292107 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.292305 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.292306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.292306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.292410 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.292627 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:33.292631 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:33.292749 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.292834 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.293018 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":130,"threads_started":1}
I20260925 02:22:33.293802 14005 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46739
I20260925 02:22:33.293892 14005 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:33.294091 14005 heartbeater.cc:511] Master 127.11.125.62:46739 requested a full tablet report, sending...
I20260925 02:22:33.294526 13515 ts_manager.cc:194] Registered new tserver with Master: 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:33.295063 13515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:55023
I20260925 02:22:33.297992 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 13872
I20260925 02:22:33.298113 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:33.298357 11764 external_mini_cluster.cc:1428] Setting key f54f90dda47bb362e40081f7ff6ff550
I20260925 02:22:33.299396 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:22:33.300762 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.294966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34039 (local address 127.11.125.62:46739)
0925 02:22:33.295015 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.295017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.295080 (+ 63us) server_negotiation.cc:407] Connection header received
0925 02:22:33.295153 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.295154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.295164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.295218 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.295605 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.296203 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.296881 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.297138 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.297188 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.297991 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.297993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.297993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.298000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.299655 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.300031 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.300031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.300032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.300091 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.300324 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.300325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.300325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.300422 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.300473 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.300644 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.300701 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:22:33.300892 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.294891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.11.125.3:34039)
0925 02:22:33.295026 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.295037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.295096 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.295241 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.295242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.295459 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:33.295523 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.295525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.296295 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.296296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.296811 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.296812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.297189 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.297190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.297250 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.297606 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:33.297610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:33.297895 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:33.299731 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.299732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.299732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.299957 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.300157 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.300158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.300158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.300256 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.300487 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:33.300488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:33.300544 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.300618 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.300739 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:33.322885 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.315690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.11.125.62:46739)
0925 02:22:33.315755 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.315758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.315768 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:33.315804 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.315805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.315814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.315882 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.316487 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.317049 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.317823 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.318020 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.318097 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.320386 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.320388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.320388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.320397 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.321695 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.322062 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.322064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.322064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.322123 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.322333 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.322335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.322335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.322454 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.322562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.322684 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.322773 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:22:33.324433 13515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:33.326072 13515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:33.332751 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.11.125.2:38869)
0925 02:22:33.328846 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.328852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.328865 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:33.328909 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.328911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.328968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.329064 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:33.329323 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330444 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.331841 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.332337 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.332427 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.332486 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.332593 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.332637 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":102,"threads_started":1}
I20260925 02:22:33.332741 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35471 (local address 127.0.0.1:38434)
0925 02:22:33.329168 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.329182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.329264 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.329533 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.329534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.329535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:33.329568 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.329570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330767 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.330768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.331685 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.331688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.332400 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.332402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.332486 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.332502 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.332577 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.332649 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":24,"thread_start_us":73,"threads_started":1}
I20260925 02:22:33.332753 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:38869 (local address 127.0.0.1:37562)
0925 02:22:33.328545 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.328586 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.328652 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.329142 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.329146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.329149 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:33.329231 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.329234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330530 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.330531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.331736 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.331739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.332427 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.332428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.332505 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.332510 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.332566 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.332641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":34,"thread_start_us":67,"threads_started":1}
I20260925 02:22:33.333194 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.11.125.1:35471)
0925 02:22:33.328945 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.328949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.329234 (+ 285us) server_negotiation.cc:407] Connection header received
0925 02:22:33.329361 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.329363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.329379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.329434 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:33.329653 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330672 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.331794 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.332302 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.332388 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.332453 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.333038 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.333090 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260925 02:22:33.333565 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.0.0.1:46462)
0925 02:22:33.328940 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.328952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.329006 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.329286 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.329287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.329289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:33.329340 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.329342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330354 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.330355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.331743 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.331745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.333311 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.333312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.333377 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.333381 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.333423 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.333469 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":10,"thread_start_us":71,"threads_started":1}
I20260925 02:22:33.333843 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.328709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.11.125.3:46123)
0925 02:22:33.329043 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.329047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.329056 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:33.329095 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.329097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.329110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.329201 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:33.329447 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.330263 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.332493 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.333197 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.333306 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.333702 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.333736 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.333772 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260925 02:22:33.333916 13797 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 (DEFAULT_TABLE table=test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.333914 13655 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 (DEFAULT_TABLE table=test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.334290 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b9c57a9e740c4950c30cc68737c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.334290 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b9c57a9e740c4950c30cc68737c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.338344 14028 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Bootstrap starting.
I20260925 02:22:33.338889 14029 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Bootstrap starting.
I20260925 02:22:33.339082 14028 tablet_bootstrap.cc:654] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.338996 13938 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 (DEFAULT_TABLE table=test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.339383 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b9c57a9e740c4950c30cc68737c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.339432 14028 log.cc:824] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:33.339514 14029 tablet_bootstrap.cc:654] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.339885 14029 log.cc:824] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:33.340428 14028 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: No bootstrap required, opened a new log
I20260925 02:22:33.340524 14028 ts_tablet_manager.cc:1397] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:33.340687 14029 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: No bootstrap required, opened a new log
I20260925 02:22:33.340771 14029 ts_tablet_manager.cc:1397] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:33.342078 14028 raft_consensus.cc:348] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.342247 14028 raft_consensus.cc:374] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.342293 14028 raft_consensus.cc:729] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.342407 14028 consensus_queue.cc:260] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.342707 14028 ts_tablet_manager.cc:1428] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:22:33.342717 13721 heartbeater.cc:503] Master 127.11.125.62:46739 was elected leader, sending a full tablet report...
I20260925 02:22:33.343016 14029 raft_consensus.cc:348] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.343181 14029 raft_consensus.cc:374] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.343247 14029 raft_consensus.cc:729] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.343364 14029 consensus_queue.cc:260] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.343654 14029 ts_tablet_manager.cc:1428] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:33.343691 13863 heartbeater.cc:503] Master 127.11.125.62:46739 was elected leader, sending a full tablet report...
I20260925 02:22:33.343873 14033 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Bootstrap starting.
I20260925 02:22:33.344707 14033 tablet_bootstrap.cc:654] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.345081 14033 log.cc:824] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:33.345816 14033 tablet_bootstrap.cc:492] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: No bootstrap required, opened a new log
I20260925 02:22:33.345902 14033 ts_tablet_manager.cc:1397] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:33.347404 14033 raft_consensus.cc:348] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.347548 14033 raft_consensus.cc:374] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.347594 14033 raft_consensus.cc:729] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.347761 14033 consensus_queue.cc:260] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.348018 14033 ts_tablet_manager.cc:1428] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:33.348104 14005 heartbeater.cc:503] Master 127.11.125.62:46739 was elected leader, sending a full tablet report...
W20260925 02:22:33.366863 13864 tablet.cc:2369] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:33.443446 14036 raft_consensus.cc:482] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:33.443603 14036 raft_consensus.cc:504] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.443975 14036 leader_election.cc:288] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471)
I20260925 02:22:33.447259 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.444453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:44531 (local address 127.11.125.1:35471)
0925 02:22:33.444505 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.444507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.444601 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:22:33.444640 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.444641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.444652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.444703 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:33.444839 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.445378 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446424 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.446883 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446974 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.446983 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.447144 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.447181 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:22:33.447278 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.444198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:38869 (local address 127.11.125.3:45967)
0925 02:22:33.444272 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.444293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.444377 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.444562 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.444563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.444564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:33.444641 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.444643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.445395 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.445396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.446235 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.446238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.446946 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.447056 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.447061 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.447140 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.447174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:22:33.447350 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.444336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35471 (local address 127.11.125.3:44531)
0925 02:22:33.444512 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.444522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.444587 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.444704 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.444705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.444706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:33.444784 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.444786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.445474 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.445475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.446306 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.446309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.446973 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.447057 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.447060 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.447136 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.447178 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":122,"threads_started":1}
I20260925 02:22:33.447506 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.444298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:45967 (local address 127.11.125.2:38869)
0925 02:22:33.444367 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.444369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.444380 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:22:33.444481 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.444483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.444494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.444544 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:33.444722 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.445300 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446332 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.446832 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.446950 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.446961 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.447386 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.447436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:22:33.447824 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b9c57a9e740c4950c30cc68737c00" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94" is_pre_election: true
I20260925 02:22:33.447835 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b9c57a9e740c4950c30cc68737c00" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c" is_pre_election: true
I20260925 02:22:33.448005 13817 raft_consensus.cc:2454] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 0.
I20260925 02:22:33.448004 13675 raft_consensus.cc:2454] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 0.
I20260925 02:22:33.448310 13889 leader_election.cc:302] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, 02dad6b1d55f410280ba8174338a2c94; no voters:
I20260925 02:22:33.448504 14036 raft_consensus.cc:2781] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:33.448684 14036 raft_consensus.cc:482] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:33.448742 14036 raft_consensus.cc:3037] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.449641 14036 raft_consensus.cc:504] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.449823 14036 leader_election.cc:288] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Requested vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471)
I20260925 02:22:33.450021 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b9c57a9e740c4950c30cc68737c00" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94"
I20260925 02:22:33.450067 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b9c57a9e740c4950c30cc68737c00" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c"
I20260925 02:22:33.450136 13675 raft_consensus.cc:3037] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.450134 13817 raft_consensus.cc:3037] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.450961 13817 raft_consensus.cc:2454] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 1.
I20260925 02:22:33.451004 13675 raft_consensus.cc:2454] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 1.
I20260925 02:22:33.451167 13889 leader_election.cc:302] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, 02dad6b1d55f410280ba8174338a2c94; no voters:
I20260925 02:22:33.451296 14036 raft_consensus.cc:2781] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:33.451503 14036 raft_consensus.cc:686] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 LEADER]: Becoming Leader. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Running, Role: LEADER
I20260925 02:22:33.451632 14036 consensus_queue.cc:237] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.452667 13516 catalog_manager.cc:5495] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3). New cstate: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } health_report { overall_health: HEALTHY } } }
W20260925 02:22:33.458173 13722 tablet.cc:2369] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:22:33 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302952, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:33.501398 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.493766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46739 (local address 127.0.0.1:51888)
0925 02:22:33.494088 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.494099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.494187 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.494366 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.494367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.494738 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:33.494793 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.494796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.495532 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.495533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.496063 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.496065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.496434 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.496435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.496499 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.497092 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:33.497097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:33.498635 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:33.500143 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.500152 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.500154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.500366 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.500603 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:33.500604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:33.500604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:33.500668 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:33.501092 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:33.501099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:33.501147 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.501214 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.501319 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":229,"threads_started":1}
I20260925 02:22:33.501518 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.493931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.11.125.62:46739)
0925 02:22:33.493997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.494000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.494164 (+ 164us) server_negotiation.cc:407] Connection header received
0925 02:22:33.494272 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.494273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.494290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.494347 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:33.494886 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.495443 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.496147 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.496341 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.496428 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.498727 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:33.498729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:33.498729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:33.498736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:33.500058 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.500437 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.500438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.500438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.500494 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:33.500751 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:33.500752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:33.500752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:33.500860 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:33.500983 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.501247 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.501386 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":33}
I20260925 02:22:33.503526 13516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:51888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:22:33.503748 13516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:33.505388 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 0967ebeafa7840d39784297425f2cdf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0421161087cc484592a4a5bf09a9bfba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:33.505445 13938 tablet_service.cc:1467] Processing CreateTablet for tablet 0967ebeafa7840d39784297425f2cdf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0421161087cc484592a4a5bf09a9bfba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:33.505435 13797 tablet_service.cc:1467] Processing CreateTablet for tablet 0967ebeafa7840d39784297425f2cdf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0421161087cc484592a4a5bf09a9bfba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:33.505606 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0967ebeafa7840d39784297425f2cdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.505606 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0967ebeafa7840d39784297425f2cdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.505628 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0967ebeafa7840d39784297425f2cdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.506862 14029 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: Bootstrap starting.
I20260925 02:22:33.507288 14028 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: Bootstrap starting.
I20260925 02:22:33.507624 14033 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: Bootstrap starting.
I20260925 02:22:33.507649 14029 tablet_bootstrap.cc:654] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.507649 14028 tablet_bootstrap.cc:654] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.508091 14033 tablet_bootstrap.cc:654] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.508389 14029 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: No bootstrap required, opened a new log
I20260925 02:22:33.508466 14029 ts_tablet_manager.cc:1397] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:33.508652 14028 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: No bootstrap required, opened a new log
I20260925 02:22:33.508690 14028 ts_tablet_manager.cc:1397] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:33.508651 14029 raft_consensus.cc:348] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.508746 14029 raft_consensus.cc:374] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.508772 14029 raft_consensus.cc:729] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.508787 14033 tablet_bootstrap.cc:492] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: No bootstrap required, opened a new log
I20260925 02:22:33.508836 14033 ts_tablet_manager.cc:1397] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:33.508823 14029 consensus_queue.cc:260] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.508867 14028 raft_consensus.cc:348] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.508936 14029 ts_tablet_manager.cc:1428] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:33.508962 14028 raft_consensus.cc:374] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.508988 14028 raft_consensus.cc:729] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.509037 14028 consensus_queue.cc:260] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.509016 14033 raft_consensus.cc:348] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.509109 14033 raft_consensus.cc:374] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.509130 14033 raft_consensus.cc:729] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.509090 14034 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.509138 14028 ts_tablet_manager.cc:1428] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:33.509166 14034 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.509169 14033 consensus_queue.cc:260] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.509194 14032 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.509275 14032 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.509289 14033 ts_tablet_manager.cc:1428] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:33.509580 14036 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.509668 14036 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 02:22:33.540643 14006 tablet.cc:2369] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:33.882310 14036 consensus_queue.cc:1035] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:33.884804 14051 raft_consensus.cc:482] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:33.884846 14038 consensus_queue.cc:1035] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:33.884882 14051 raft_consensus.cc:504] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.885069 14051 leader_election.cc:288] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869)
I20260925 02:22:33.885218 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967ebeafa7840d39784297425f2cdf2" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c" is_pre_election: true
I20260925 02:22:33.885207 13816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967ebeafa7840d39784297425f2cdf2" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94" is_pre_election: true
I20260925 02:22:33.885320 13675 raft_consensus.cc:2454] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 0.
I20260925 02:22:33.885322 13816 raft_consensus.cc:2454] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 0.
I20260925 02:22:33.885454 13889 leader_election.cc:302] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:33.885751 14051 raft_consensus.cc:2781] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:33.885802 14051 raft_consensus.cc:482] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:33.885833 14051 raft_consensus.cc:3037] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.886720 14051 raft_consensus.cc:504] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.886870 14051 leader_election.cc:288] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Requested vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869)
I20260925 02:22:33.887084 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967ebeafa7840d39784297425f2cdf2" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c"
I20260925 02:22:33.887104 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0967ebeafa7840d39784297425f2cdf2" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94"
I20260925 02:22:33.887173 13675 raft_consensus.cc:3037] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.887193 13817 raft_consensus.cc:3037] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:33.887756 13817 raft_consensus.cc:2454] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 1.
I20260925 02:22:33.887923 13889 leader_election.cc:302] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, 02dad6b1d55f410280ba8174338a2c94; no voters:
I20260925 02:22:33.887933 13675 raft_consensus.cc:2454] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 1.
I20260925 02:22:33.888027 14051 raft_consensus.cc:2781] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:33.888077 14051 raft_consensus.cc:686] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 LEADER]: Becoming Leader. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Running, Role: LEADER
I20260925 02:22:33.888127 14051 consensus_queue.cc:237] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.888399 14038 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: TxnStatusTablet state changed. Reason: New leader 009ea5b089a3465cbcd0405f182a8a7e. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.888496 14038 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:33.888669 14056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:22:33.888636 13515 catalog_manager.cc:5495] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3). New cstate: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } health_report { overall_health: HEALTHY } } }
I20260925 02:22:33.888777 14056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:22:33.952868 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.949686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.11.125.3:46123)
0925 02:22:33.949843 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.949846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.949854 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:33.949891 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.949892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.949911 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.950015 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:33.950192 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.950719 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.951987 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.952195 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.952259 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.952513 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:33.952616 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:33.952694 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.952778 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.952800 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260925 02:22:33.952945 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.949513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.0.0.1:46472)
0925 02:22:33.949598 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.949612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.949676 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.950030 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.950032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.950033 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:33.950110 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.950113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.950854 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.950855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.951889 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.951893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.952285 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.952286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.952364 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.952384 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:33.952700 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:33.952710 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.952769 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.952825 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:22:33.954689 13817 raft_consensus.cc:1261] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Refusing update from remote peer 009ea5b089a3465cbcd0405f182a8a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:33.954689 13675 raft_consensus.cc:1261] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Refusing update from remote peer 009ea5b089a3465cbcd0405f182a8a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:33.954936 14038 consensus_queue.cc:1035] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:33.955049 14051 consensus_queue.cc:1035] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:33.956156 14032 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: TxnStatusTablet state changed. Reason: New leader 009ea5b089a3465cbcd0405f182a8a7e. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.956173 14034 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: TxnStatusTablet state changed. Reason: New leader 009ea5b089a3465cbcd0405f182a8a7e. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.956267 14032 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.956276 14034 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.956727 14051 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.956802 14051 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:33.956979 14050 mvcc.cc:205] Tried to move back new op lower bound from 7333080899396001792 to 7333080899126185984. Current Snapshot: MvccSnapshot[applied={T|T < 7333080899396001792}]
I20260925 02:22:33.958228 14036 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.958338 14036 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 009ea5b089a3465cbcd0405f182a8a7e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:33.959005 14034 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.959098 14034 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P 02dad6b1d55f410280ba8174338a2c94: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.959611 14052 mvcc.cc:205] Tried to move back new op lower bound from 7333080899396001792 to 7333080899126185984. Current Snapshot: MvccSnapshot[applied={T|T < 7333080899396001792}]
I20260925 02:22:33.959702 14032 tablet_replica.cc:439] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "009ea5b089a3465cbcd0405f182a8a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } } }
I20260925 02:22:33.959761 14032 tablet_replica.cc:442] T 0967ebeafa7840d39784297425f2cdf2 P a4b6d136b239470eb93776872a9a472c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:33.959995 14054 mvcc.cc:205] Tried to move back new op lower bound from 7333080899396001792 to 7333080899126185984. Current Snapshot: MvccSnapshot[applied={T|T < 7333080899396001792 or (T in {7333080899396001792})}]
I20260925 02:22:33.964890 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.962153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.11.125.3:46123)
0925 02:22:33.962199 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.962201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.962317 (+ 116us) server_negotiation.cc:407] Connection header received
0925 02:22:33.962391 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.962392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.962400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.962445 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:33.962580 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.963049 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.964268 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.964449 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.964501 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.964678 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:33.964705 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:33.964742 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.964798 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.964816 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:33.970121 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.967658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:46629 (local address 127.11.125.3:46123)
0925 02:22:33.967695 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:33.967697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:33.967840 (+ 143us) server_negotiation.cc:407] Connection header received
0925 02:22:33.967901 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:33.967902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:33.967913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:33.967961 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:33.968104 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.968551 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.969398 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.969582 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.969655 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.969840 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:33.969876 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:33.969939 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:33.969998 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:22:33.970033 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":19}
I20260925 02:22:33.970367 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:33.967580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.11.125.3:46629)
0925 02:22:33.967773 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:33.967781 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:33.967855 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:33.967962 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:33.967963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:33.967964 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:33.968032 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.968033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.968635 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.968636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.969300 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:33.969303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:33.969654 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:33.969654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:33.969721 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:33.969734 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:33.969941 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:33.969943 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:33.969995 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:22:33.970052 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260925 02:22:33.987026 13514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:33.987182 13514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:33.988130 13514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Sending DeleteTablet for 3 replicas of tablet ad2b9c57a9e740c4950c30cc68737c00
I20260925 02:22:33.988482 13655 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:33 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38434
I20260925 02:22:33.988495 13797 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:33 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260925 02:22:33.988770 13937 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b9c57a9e740c4950c30cc68737c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:33 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260925 02:22:33.989077 14076 tablet_replica.cc:330] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: stopping tablet replica
I20260925 02:22:33.989189 14074 tablet_replica.cc:330] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: stopping tablet replica
I20260925 02:22:33.989287 14076 raft_consensus.cc:2229] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:33.989300 14074 raft_consensus.cc:2229] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:33.989362 14074 raft_consensus.cc:2258] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:33.989322 13514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:33.989482 13514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:33.989792 14074 ts_tablet_manager.cc:1905] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:33.989934 14076 raft_consensus.cc:2258] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:33.990350 14076 ts_tablet_manager.cc:1905] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:33.990856 13937 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd7796a16e44787b02078e472b6ad44 (DEFAULT_TABLE table=test-table [id=f113d3c9ed974262b00df8ec833712f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.990968 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd7796a16e44787b02078e472b6ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.991088 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd7796a16e44787b02078e472b6ad44 (DEFAULT_TABLE table=test-table [id=f113d3c9ed974262b00df8ec833712f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.991215 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd7796a16e44787b02078e472b6ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.991699 14074 ts_tablet_manager.cc:1918] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:33.991770 14074 log.cc:1196] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/ad2b9c57a9e740c4950c30cc68737c00
I20260925 02:22:33.992034 14074 ts_tablet_manager.cc:1939] T ad2b9c57a9e740c4950c30cc68737c00 P a4b6d136b239470eb93776872a9a472c: Deleting consensus metadata
I20260925 02:22:33.989521 14075 tablet_replica.cc:330] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: stopping tablet replica
I20260925 02:22:33.992303 14075 raft_consensus.cc:2229] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:33.992365 14075 raft_consensus.cc:2258] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:33.992583 14033 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Bootstrap starting.
I20260925 02:22:33.993142 14033 tablet_bootstrap.cc:654] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.993534 14028 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Bootstrap starting.
I20260925 02:22:33.993904 14076 ts_tablet_manager.cc:1918] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:33.993966 14028 tablet_bootstrap.cc:654] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.993992 14076 log.cc:1196] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/ad2b9c57a9e740c4950c30cc68737c00
I20260925 02:22:33.994297 14076 ts_tablet_manager.cc:1939] T ad2b9c57a9e740c4950c30cc68737c00 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting consensus metadata
I20260925 02:22:33.990863 13797 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd7796a16e44787b02078e472b6ad44 (DEFAULT_TABLE table=test-table [id=f113d3c9ed974262b00df8ec833712f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:33.994597 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd7796a16e44787b02078e472b6ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:33.994702 13502 catalog_manager.cc:4841] TS 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123): tablet ad2b9c57a9e740c4950c30cc68737c00 (table test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]) successfully deleted
I20260925 02:22:33.996018 14029 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Bootstrap starting.
I20260925 02:22:33.996449 14029 tablet_bootstrap.cc:654] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:33.996459 13501 catalog_manager.cc:4841] TS a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471): tablet ad2b9c57a9e740c4950c30cc68737c00 (table test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]) successfully deleted
I20260925 02:22:33.996467 14075 ts_tablet_manager.cc:1905] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:33.996724 14028 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: No bootstrap required, opened a new log
I20260925 02:22:33.996786 14028 ts_tablet_manager.cc:1397] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:22:33.996970 14028 raft_consensus.cc:348] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.997068 14029 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: No bootstrap required, opened a new log
I20260925 02:22:33.997071 14028 raft_consensus.cc:374] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.997098 14028 raft_consensus.cc:729] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.997107 14029 ts_tablet_manager.cc:1397] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:33.997149 14028 consensus_queue.cc:260] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.997244 14028 ts_tablet_manager.cc:1428] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:33.997241 14029 raft_consensus.cc:348] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.997298 14029 raft_consensus.cc:374] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.997316 14029 raft_consensus.cc:729] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.997356 14029 consensus_queue.cc:260] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.997443 14029 ts_tablet_manager.cc:1428] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:33.998188 14033 tablet_bootstrap.cc:492] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: No bootstrap required, opened a new log
I20260925 02:22:33.998251 14033 ts_tablet_manager.cc:1397] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260925 02:22:33.998390 14033 raft_consensus.cc:348] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.998472 14033 raft_consensus.cc:374] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:33.998497 14033 raft_consensus.cc:729] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Initialized, Role: FOLLOWER
I20260925 02:22:33.998550 14033 consensus_queue.cc:260] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:33.998668 14033 ts_tablet_manager.cc:1428] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:33.998831 14075 ts_tablet_manager.cc:1918] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:33.998919 14075 log.cc:1196] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/ad2b9c57a9e740c4950c30cc68737c00
I20260925 02:22:33.999197 14075 ts_tablet_manager.cc:1939] T ad2b9c57a9e740c4950c30cc68737c00 P 02dad6b1d55f410280ba8174338a2c94: Deleting consensus metadata
I20260925 02:22:33.999668 13501 catalog_manager.cc:4841] TS 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869): tablet ad2b9c57a9e740c4950c30cc68737c00 (table test-table [id=7cd7ec5586da416e92e6e31ff0d9621f]) successfully deleted
I20260925 02:22:34.018646 14032 raft_consensus.cc:482] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:34.018801 14032 raft_consensus.cc:504] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:34.019167 14032 leader_election.cc:288] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:34.023057 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.019494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42715 (local address 127.11.125.3:46123)
0925 02:22:34.019546 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:34.019548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:34.019656 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:22:34.019805 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:34.019806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:34.019815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:34.019876 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:34.020028 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.020782 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.022094 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.022646 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.022731 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.022742 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:34.022930 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:22:34.022986 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:22:34.023063 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.019366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.11.125.1:42715)
0925 02:22:34.019599 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:34.019610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:34.019689 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:34.019881 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:34.019882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:34.019884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:34.019952 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.019954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.020876 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.020877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.021999 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.022002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.022745 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.022746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.022822 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.022828 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:22:34.022902 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:22:34.022958 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":106,"thread_start_us":73,"threads_started":1}
I20260925 02:22:34.023272 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.019356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:38869 (local address 127.11.125.1:53957)
0925 02:22:34.019574 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:34.019587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:34.019646 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:34.019938 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:34.019940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:34.019942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:34.020042 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.020044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.021047 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.021048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.022168 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.022172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.023000 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.023001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.023075 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.023079 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:34.023128 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:34.023169 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260925 02:22:34.023509 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.019567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:53957 (local address 127.11.125.2:38869)
0925 02:22:34.019710 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:34.019713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:34.019723 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:34.019778 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:34.019779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:34.019791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:34.019877 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:34.020128 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.020950 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.022279 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.022862 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.022950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.022963 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:34.023367 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 02:22:34.023420 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":50,"threads_started":1}
I20260925 02:22:34.023579 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd7796a16e44787b02078e472b6ad44" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e" is_pre_election: true
I20260925 02:22:34.023751 13958 raft_consensus.cc:2454] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 0.
I20260925 02:22:34.023753 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd7796a16e44787b02078e472b6ad44" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94" is_pre_election: true
I20260925 02:22:34.023826 13817 raft_consensus.cc:2454] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 0.
I20260925 02:22:34.024003 13607 leader_election.cc:302] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:34.024190 14032 raft_consensus.cc:2781] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:34.024259 14032 raft_consensus.cc:482] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:34.024291 14032 raft_consensus.cc:3037] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:34.025063 14032 raft_consensus.cc:504] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:34.025211 14032 leader_election.cc:288] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 election: Requested vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:34.025367 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd7796a16e44787b02078e472b6ad44" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94"
I20260925 02:22:34.025471 13817 raft_consensus.cc:3037] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:34.026028 13817 raft_consensus.cc:2454] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 1.
I20260925 02:22:34.026229 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd7796a16e44787b02078e472b6ad44" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e"
I20260925 02:22:34.026276 13606 leader_election.cc:302] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:34.026326 13958 raft_consensus.cc:3037] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:34.026420 14032 raft_consensus.cc:2781] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:34.026610 14032 raft_consensus.cc:686] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 1 LEADER]: Becoming Leader. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Running, Role: LEADER
I20260925 02:22:34.026697 14032 consensus_queue.cc:237] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:34.027232 13958 raft_consensus.cc:2454] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 1.
I20260925 02:22:34.027519 13515 catalog_manager.cc:5495] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b6d136b239470eb93776872a9a472c (127.11.125.1). New cstate: current_term: 1 leader_uuid: "a4b6d136b239470eb93776872a9a472c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:34.043099 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.040128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.11.125.1:35471)
0925 02:22:34.040291 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:34.040293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:34.040301 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:34.040345 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:34.040345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:34.040364 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:34.040433 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:34.040591 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.041279 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.042355 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.042552 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.042603 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.042780 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:34.042882 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:34.042937 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:34.043008 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 02:22:34.043026 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260925 02:22:34.049424 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.046211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:45867 (local address 127.11.125.3:46123)
0925 02:22:34.046282 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:34.046284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:34.046293 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:34.046352 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:34.046354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:34.046366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:34.046440 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:34.046528 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.047318 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.048636 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.048938 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.049007 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.049165 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:34.049199 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:34.049248 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:34.049320 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:22:34.049341 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:22:34.049497 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.046071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.11.125.1:45867)
0925 02:22:34.046224 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:34.046236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:34.046288 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:34.046441 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:34.046442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:34.046443 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:34.046490 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.046491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.047420 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.047421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.048546 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.048549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.049028 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.049029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.049075 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.049090 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:34.049265 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:34.049268 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:34.049317 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:34.049354 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260925 02:22:34.051712 13958 raft_consensus.cc:1261] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Refusing update from remote peer a4b6d136b239470eb93776872a9a472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:34.051718 13817 raft_consensus.cc:1261] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Refusing update from remote peer a4b6d136b239470eb93776872a9a472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:34.052012 14083 consensus_queue.cc:1035] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:34.052152 14032 consensus_queue.cc:1035] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:34.059975 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.057379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35471 (local address 127.11.125.3:43977)
0925 02:22:34.057475 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:34.057486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:34.057528 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:34.057676 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:34.057676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:34.057678 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:34.057729 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.057730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.058372 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.058373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.059238 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:34.059242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.059585 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.059586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:34.059643 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.059650 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:34.059806 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:34.059808 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:34.059852 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 02:22:34.059895 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:22:34.060184 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:34.057482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:43977 (local address 127.11.125.1:35471)
0925 02:22:34.057521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:34.057523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:34.057531 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:34.057606 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:34.057607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:34.057618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:34.057660 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:34.057801 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.058287 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.059322 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:34.059522 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:34.059565 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:34.059709 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:34.059745 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:34.059799 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:34.060096 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 02:22:34.060120 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:22:34.060539 14052 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:22:35.586546 13515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:35.586679 13515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:35.587625 13515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Sending DeleteTablet for 3 replicas of tablet 1dd7796a16e44787b02078e472b6ad44
I20260925 02:22:35.587934 13937 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd7796a16e44787b02078e472b6ad44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260925 02:22:35.587992 13655 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd7796a16e44787b02078e472b6ad44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38434
I20260925 02:22:35.588003 13797 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd7796a16e44787b02078e472b6ad44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260925 02:22:35.588222 14116 tablet_replica.cc:330] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: stopping tablet replica
I20260925 02:22:35.588281 14116 raft_consensus.cc:2229] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:35.588321 14116 raft_consensus.cc:2258] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.588598 14116 ts_tablet_manager.cc:1905] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.589079 14114 tablet_replica.cc:330] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: stopping tablet replica
I20260925 02:22:35.589166 14114 raft_consensus.cc:2229] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:35.589229 14114 raft_consensus.cc:2258] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.589360 14115 tablet_replica.cc:330] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: stopping tablet replica
I20260925 02:22:35.589440 14115 raft_consensus.cc:2229] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:35.589583 14115 raft_consensus.cc:2258] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.590975 14115 ts_tablet_manager.cc:1905] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.591029 14114 ts_tablet_manager.cc:1905] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.591152 14116 ts_tablet_manager.cc:1918] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:22:35.591310 14116 log.cc:1196] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/1dd7796a16e44787b02078e472b6ad44
I20260925 02:22:35.591696 14116 ts_tablet_manager.cc:1939] T 1dd7796a16e44787b02078e472b6ad44 P 02dad6b1d55f410280ba8174338a2c94: Deleting consensus metadata
I20260925 02:22:35.592353 13501 catalog_manager.cc:4841] TS 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869): tablet 1dd7796a16e44787b02078e472b6ad44 (table test-table [id=f113d3c9ed974262b00df8ec833712f2]) successfully deleted
I20260925 02:22:35.592770 14114 ts_tablet_manager.cc:1918] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:22:35.592867 14114 log.cc:1196] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/1dd7796a16e44787b02078e472b6ad44
I20260925 02:22:35.592882 14115 ts_tablet_manager.cc:1918] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:22:35.592973 14115 log.cc:1196] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/1dd7796a16e44787b02078e472b6ad44
I20260925 02:22:35.593114 14114 ts_tablet_manager.cc:1939] T 1dd7796a16e44787b02078e472b6ad44 P 009ea5b089a3465cbcd0405f182a8a7e: Deleting consensus metadata
I20260925 02:22:35.593194 14115 ts_tablet_manager.cc:1939] T 1dd7796a16e44787b02078e472b6ad44 P a4b6d136b239470eb93776872a9a472c: Deleting consensus metadata
I20260925 02:22:35.593801 13502 catalog_manager.cc:4841] TS 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123): tablet 1dd7796a16e44787b02078e472b6ad44 (table test-table [id=f113d3c9ed974262b00df8ec833712f2]) successfully deleted
I20260925 02:22:35.593925 13501 catalog_manager.cc:4841] TS a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471): tablet 1dd7796a16e44787b02078e472b6ad44 (table test-table [id=f113d3c9ed974262b00df8ec833712f2]) successfully deleted
Sep 25 02:22:35 dist-test-slave-hj7s krb5kdc[13471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:22:35 dist-test-slave-hj7s krb5kdc[13471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:35.614672 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.607091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.11.125.62:46739)
0925 02:22:35.607282 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.607284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.607296 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:35.607391 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.607392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.607405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.607499 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:35.608092 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.608621 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.609467 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.609678 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.609771 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.612001 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:35.612003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:35.612004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:35.612012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:35.613459 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:35.613834 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:35.613836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:35.613836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:35.613900 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:35.614109 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:35.614110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:35.614111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:35.614231 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:35.614347 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.614466 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.614554 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260925 02:22:35.616050 13515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:35.616207 13515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:35.617647 13655 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa9ed212bc41218dd70ce58126f14d (DEFAULT_TABLE table=test-table [id=917ff17397754eee9f38b8eb0794a30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.617647 13937 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa9ed212bc41218dd70ce58126f14d (DEFAULT_TABLE table=test-table [id=917ff17397754eee9f38b8eb0794a30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.617662 13797 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa9ed212bc41218dd70ce58126f14d (DEFAULT_TABLE table=test-table [id=917ff17397754eee9f38b8eb0794a30c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.617797 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa9ed212bc41218dd70ce58126f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.617797 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa9ed212bc41218dd70ce58126f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.617818 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa9ed212bc41218dd70ce58126f14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.618860 14125 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Bootstrap starting.
I20260925 02:22:35.619247 14126 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Bootstrap starting.
I20260925 02:22:35.619382 14125 tablet_bootstrap.cc:654] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.619381 14127 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Bootstrap starting.
I20260925 02:22:35.619724 14126 tablet_bootstrap.cc:654] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.619772 14127 tablet_bootstrap.cc:654] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.620240 14125 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: No bootstrap required, opened a new log
I20260925 02:22:35.620303 14125 ts_tablet_manager.cc:1397] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:35.620343 14126 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: No bootstrap required, opened a new log
I20260925 02:22:35.620391 14126 ts_tablet_manager.cc:1397] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:35.620396 14127 tablet_bootstrap.cc:492] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: No bootstrap required, opened a new log
I20260925 02:22:35.620446 14127 ts_tablet_manager.cc:1397] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:35.620429 14125 raft_consensus.cc:348] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.620499 14125 raft_consensus.cc:374] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.620532 14125 raft_consensus.cc:729] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.620497 14126 raft_consensus.cc:348] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.620580 14126 raft_consensus.cc:374] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.620607 14126 raft_consensus.cc:729] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.620576 14125 consensus_queue.cc:260] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.620599 14127 raft_consensus.cc:348] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.620682 14127 raft_consensus.cc:374] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.620651 14126 consensus_queue.cc:260] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.620713 14127 raft_consensus.cc:729] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.620761 14125 ts_tablet_manager.cc:1428] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:35.620761 14126 ts_tablet_manager.cc:1428] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:35.620795 14127 consensus_queue.cc:260] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.621057 14127 ts_tablet_manager.cc:1428] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:35.866647 14131 raft_consensus.cc:482] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:35.866753 14131 raft_consensus.cc:504] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.867208 14131 leader_election.cc:288] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:35.870708 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.867465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:51665 (local address 127.11.125.1:35471)
0925 02:22:35.867646 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.867648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.867879 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:22:35.867911 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.867912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.867923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.867999 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:35.868194 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.868688 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.869696 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.870293 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.870380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.870394 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.870609 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.870647 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260925 02:22:35.870728 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.867416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35471 (local address 127.11.125.2:51665)
0925 02:22:35.867771 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:35.867782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:35.867853 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:35.868014 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:35.868015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:35.868017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:35.868113 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.868115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.868783 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.868784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.869599 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.869601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.870380 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.870403 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.870514 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.870519 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:35.870584 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:22:35.870627 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":26,"thread_start_us":134,"threads_started":1}
I20260925 02:22:35.871095 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "02dad6b1d55f410280ba8174338a2c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c" is_pre_election: true
I20260925 02:22:35.871209 13675 raft_consensus.cc:2454] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02dad6b1d55f410280ba8174338a2c94 in term 0.
I20260925 02:22:35.871444 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.867411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.11.125.2:37307)
0925 02:22:35.867624 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:35.867637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:35.867704 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:35.867898 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:35.867899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:35.867902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:35.867991 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.867993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.868723 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.868724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.870347 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.870351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.871129 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.871131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.871203 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.871206 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:35.871295 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:22:35.871345 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":64,"thread_start_us":131,"threads_started":1}
I20260925 02:22:35.871481 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.867464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:37307 (local address 127.11.125.3:46123)
0925 02:22:35.867661 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.867663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.867683 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:22:35.867786 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.867787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.867802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.867873 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:35.868073 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.868633 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.870437 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.871058 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.871131 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.871140 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.871362 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.871414 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260925 02:22:35.871412 13748 leader_election.cc:302] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:35.871580 14131 raft_consensus.cc:2781] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:35.871618 14131 raft_consensus.cc:482] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:35.871632 14131 raft_consensus.cc:3037] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:35.871693 14071 raft_consensus.cc:482] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:35.871762 14071 raft_consensus.cc:504] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.871784 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "02dad6b1d55f410280ba8174338a2c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e" is_pre_election: true
I20260925 02:22:35.871862 13958 raft_consensus.cc:2454] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02dad6b1d55f410280ba8174338a2c94 in term 0.
I20260925 02:22:35.871896 14071 leader_election.cc:288] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869)
I20260925 02:22:35.872093 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c" is_pre_election: true
I20260925 02:22:35.872138 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94" is_pre_election: true
I20260925 02:22:35.872176 13675 raft_consensus.cc:2454] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009ea5b089a3465cbcd0405f182a8a7e in term 0.
I20260925 02:22:35.872316 13889 leader_election.cc:302] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:35.872408 14071 raft_consensus.cc:2781] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:35.872457 14071 raft_consensus.cc:482] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:35.872490 14071 raft_consensus.cc:3037] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:35.872665 14131 raft_consensus.cc:504] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.872800 14131 leader_election.cc:288] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [CANDIDATE]: Term 1 election: Requested vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:35.872822 13817 raft_consensus.cc:2379] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009ea5b089a3465cbcd0405f182a8a7e in current term 1: Already voted for candidate 02dad6b1d55f410280ba8174338a2c94 in this term.
I20260925 02:22:35.872990 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "02dad6b1d55f410280ba8174338a2c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c"
I20260925 02:22:35.873006 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "02dad6b1d55f410280ba8174338a2c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e"
I20260925 02:22:35.873065 13675 raft_consensus.cc:3037] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:35.873414 14071 raft_consensus.cc:504] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.873550 14071 leader_election.cc:288] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Requested vote from peers a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471), 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869)
I20260925 02:22:35.873589 13958 raft_consensus.cc:2379] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02dad6b1d55f410280ba8174338a2c94 in current term 1: Already voted for candidate 009ea5b089a3465cbcd0405f182a8a7e in this term.
I20260925 02:22:35.873714 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94"
I20260925 02:22:35.873829 13817 raft_consensus.cc:2379] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 009ea5b089a3465cbcd0405f182a8a7e in current term 1: Already voted for candidate 02dad6b1d55f410280ba8174338a2c94 in this term.
I20260925 02:22:35.873958 13675 raft_consensus.cc:2454] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02dad6b1d55f410280ba8174338a2c94 in term 1.
I20260925 02:22:35.874107 13748 leader_election.cc:302] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c; no voters: 009ea5b089a3465cbcd0405f182a8a7e
I20260925 02:22:35.874140 13675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa9ed212bc41218dd70ce58126f14d" candidate_uuid: "009ea5b089a3465cbcd0405f182a8a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b6d136b239470eb93776872a9a472c"
I20260925 02:22:35.874262 13675 raft_consensus.cc:2379] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 009ea5b089a3465cbcd0405f182a8a7e in current term 1: Already voted for candidate 02dad6b1d55f410280ba8174338a2c94 in this term.
I20260925 02:22:35.874271 14131 raft_consensus.cc:2781] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:35.874430 14131 raft_consensus.cc:686] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 LEADER]: Becoming Leader. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Running, Role: LEADER
I20260925 02:22:35.874519 14131 consensus_queue.cc:237] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.874971 13889 leader_election.cc:302] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009ea5b089a3465cbcd0405f182a8a7e; no voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c
I20260925 02:22:35.875067 14071 raft_consensus.cc:2726] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:22:35.875190 13516 catalog_manager.cc:5495] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2). New cstate: current_term: 1 leader_uuid: "02dad6b1d55f410280ba8174338a2c94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:35.909109 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.905348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.11.125.2:38869)
0925 02:22:35.905554 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.905557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.905566 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:35.905616 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.905617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.905642 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.905728 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:35.905935 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.906773 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.908086 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.908386 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.908458 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.908688 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:35.908834 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:35.908941 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.908988 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.909015 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":129,"threads_started":1}
I20260925 02:22:35.919477 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.916660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:39683 (local address 127.11.125.3:46123)
0925 02:22:35.916759 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.916761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.917062 (+ 301us) server_negotiation.cc:407] Connection header received
0925 02:22:35.917145 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.917146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.917159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.917211 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:35.917365 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.917852 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.918707 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.918925 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.919012 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.919203 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:35.919236 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:35.919304 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.919374 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.919401 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:22:35.919528 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.916491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:46123 (local address 127.11.125.2:39683)
0925 02:22:35.916979 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:35.916991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:35.917080 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:35.917211 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:35.917212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:35.917214 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:35.917284 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.917286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.917940 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.917942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.918539 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.918541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.919014 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.919015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.919090 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.919111 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:35.919306 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:35.919309 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:35.919369 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:22:35.919434 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":90,"threads_started":1}
I20260925 02:22:35.921823 13958 raft_consensus.cc:1261] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Refusing update from remote peer 02dad6b1d55f410280ba8174338a2c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:35.921823 13675 raft_consensus.cc:1261] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Refusing update from remote peer 02dad6b1d55f410280ba8174338a2c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:35.922097 14136 consensus_queue.cc:1035] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:35.922246 14131 consensus_queue.cc:1035] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:35.929956 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.927541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34541 (local address 127.11.125.2:38869)
0925 02:22:35.927584 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:35.927587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:35.927692 (+ 105us) server_negotiation.cc:407] Connection header received
0925 02:22:35.927745 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:35.927745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:35.927755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:35.927803 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:35.927887 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.928341 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.929235 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.929557 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.929623 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.929778 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:35.929805 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:35.929845 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:35.929882 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:22:35.929900 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:22:35.929966 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:35.927486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:38869 (local address 127.11.125.3:34541)
0925 02:22:35.927617 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:35.927627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:35.927691 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:35.927809 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:35.927810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:35.927811 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:35.927853 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.927855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.928405 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.928406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.929149 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:35.929152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:35.929631 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:35.929632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:35.929695 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:35.929701 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:35.929844 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:35.929845 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:22:35.929873 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 02:22:35.929907 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 02:22:35.936728 13516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:35.936853 13516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:35.937556 13516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Sending DeleteTablet for 3 replicas of tablet d0fa9ed212bc41218dd70ce58126f14d
I20260925 02:22:35.937826 13937 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fa9ed212bc41218dd70ce58126f14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260925 02:22:35.937875 13797 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fa9ed212bc41218dd70ce58126f14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260925 02:22:35.937927 14114 tablet_replica.cc:330] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: stopping tablet replica
I20260925 02:22:35.937963 14116 tablet_replica.cc:330] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: stopping tablet replica
I20260925 02:22:35.937981 14114 raft_consensus.cc:2229] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:35.938020 14114 raft_consensus.cc:2258] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.938019 14116 raft_consensus.cc:2229] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:35.938107 14116 raft_consensus.cc:2258] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.938189 13655 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fa9ed212bc41218dd70ce58126f14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:35 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38434
I20260925 02:22:35.938339 14115 tablet_replica.cc:330] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: stopping tablet replica
I20260925 02:22:35.938402 14114 ts_tablet_manager.cc:1905] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.938429 14115 raft_consensus.cc:2229] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:35.938469 14115 raft_consensus.cc:2258] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:35.938715 14116 ts_tablet_manager.cc:1905] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.938674 13516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:35.938815 13516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:35.939157 14115 ts_tablet_manager.cc:1905] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:35.940248 14114 ts_tablet_manager.cc:1918] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:35.940244 14116 ts_tablet_manager.cc:1918] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:35.940305 14116 log.cc:1196] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/d0fa9ed212bc41218dd70ce58126f14d
I20260925 02:22:35.940305 14114 log.cc:1196] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/d0fa9ed212bc41218dd70ce58126f14d
I20260925 02:22:35.940528 14116 ts_tablet_manager.cc:1939] T d0fa9ed212bc41218dd70ce58126f14d P 02dad6b1d55f410280ba8174338a2c94: Deleting consensus metadata
I20260925 02:22:35.940529 14114 ts_tablet_manager.cc:1939] T d0fa9ed212bc41218dd70ce58126f14d P 009ea5b089a3465cbcd0405f182a8a7e: Deleting consensus metadata
I20260925 02:22:35.940853 14115 ts_tablet_manager.cc:1918] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:35.940936 14115 log.cc:1196] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/d0fa9ed212bc41218dd70ce58126f14d
I20260925 02:22:35.941076 13502 catalog_manager.cc:4841] TS 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123): tablet d0fa9ed212bc41218dd70ce58126f14d (table test-table [id=917ff17397754eee9f38b8eb0794a30c]) successfully deleted
I20260925 02:22:35.941148 14115 ts_tablet_manager.cc:1939] T d0fa9ed212bc41218dd70ce58126f14d P a4b6d136b239470eb93776872a9a472c: Deleting consensus metadata
I20260925 02:22:35.941540 13501 catalog_manager.cc:4841] TS 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869): tablet d0fa9ed212bc41218dd70ce58126f14d (table test-table [id=917ff17397754eee9f38b8eb0794a30c]) successfully deleted
I20260925 02:22:35.941614 13501 catalog_manager.cc:4841] TS a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471): tablet d0fa9ed212bc41218dd70ce58126f14d (table test-table [id=917ff17397754eee9f38b8eb0794a30c]) successfully deleted
I20260925 02:22:35.942081 13655 tablet_service.cc:1467] Processing CreateTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad (DEFAULT_TABLE table=test-table [id=b2a7e2f43b31489d962ba87445db26f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.942215 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.943046 14126 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c: Bootstrap starting.
I20260925 02:22:35.943104 13797 tablet_service.cc:1467] Processing CreateTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad (DEFAULT_TABLE table=test-table [id=b2a7e2f43b31489d962ba87445db26f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.943205 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.943401 14126 tablet_bootstrap.cc:654] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.943678 13937 tablet_service.cc:1467] Processing CreateTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad (DEFAULT_TABLE table=test-table [id=b2a7e2f43b31489d962ba87445db26f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:35.943799 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:35.944480 14125 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Bootstrap starting.
I20260925 02:22:35.944972 14125 tablet_bootstrap.cc:654] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.945140 14127 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Bootstrap starting.
I20260925 02:22:35.945477 14127 tablet_bootstrap.cc:654] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:35.945492 14126 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c: No bootstrap required, opened a new log
I20260925 02:22:35.945533 14126 ts_tablet_manager.cc:1397] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:22:35.945674 14126 raft_consensus.cc:348] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.945755 14126 raft_consensus.cc:374] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.945776 14126 raft_consensus.cc:729] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.945853 14126 consensus_queue.cc:260] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.945981 14126 ts_tablet_manager.cc:1428] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:35.946137 14127 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: No bootstrap required, opened a new log
I20260925 02:22:35.946189 14127 ts_tablet_manager.cc:1397] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:35.946276 14125 tablet_bootstrap.cc:492] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: No bootstrap required, opened a new log
I20260925 02:22:35.946326 14125 ts_tablet_manager.cc:1397] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:22:35.946348 14127 raft_consensus.cc:348] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.946455 14127 raft_consensus.cc:374] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.946476 14127 raft_consensus.cc:729] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009ea5b089a3465cbcd0405f182a8a7e, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.946491 14125 raft_consensus.cc:348] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.946519 14127 consensus_queue.cc:260] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.946556 14125 raft_consensus.cc:374] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:35.946592 14125 raft_consensus.cc:729] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dad6b1d55f410280ba8174338a2c94, State: Initialized, Role: FOLLOWER
I20260925 02:22:35.946612 14127 ts_tablet_manager.cc:1428] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:35.946635 14125 consensus_queue.cc:260] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:35.946750 14125 ts_tablet_manager.cc:1428] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:36.209681 14108 raft_consensus.cc:482] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:36.209826 14108 raft_consensus.cc:504] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:36.210062 14108 leader_election.cc:288] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:36.210320 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7173f7bc8f9d4b3ab7b71295fe9341ad" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94" is_pre_election: true
I20260925 02:22:36.210328 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7173f7bc8f9d4b3ab7b71295fe9341ad" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e" is_pre_election: true
I20260925 02:22:36.210433 13958 raft_consensus.cc:2454] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 0.
I20260925 02:22:36.210433 13817 raft_consensus.cc:2454] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 0.
I20260925 02:22:36.210636 13606 leader_election.cc:302] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:36.210791 14108 raft_consensus.cc:2781] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:36.210842 14108 raft_consensus.cc:482] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:36.210857 14108 raft_consensus.cc:3037] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:36.211589 14108 raft_consensus.cc:504] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:36.211772 14108 leader_election.cc:288] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 election: Requested vote from peers 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869), 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123)
I20260925 02:22:36.211951 13817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7173f7bc8f9d4b3ab7b71295fe9341ad" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dad6b1d55f410280ba8174338a2c94"
I20260925 02:22:36.211952 13958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7173f7bc8f9d4b3ab7b71295fe9341ad" candidate_uuid: "a4b6d136b239470eb93776872a9a472c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009ea5b089a3465cbcd0405f182a8a7e"
I20260925 02:22:36.212041 13958 raft_consensus.cc:3037] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:36.212054 13817 raft_consensus.cc:3037] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:36.212679 13817 raft_consensus.cc:2454] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 1.
I20260925 02:22:36.212690 13958 raft_consensus.cc:2454] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b6d136b239470eb93776872a9a472c in term 1.
I20260925 02:22:36.212867 13606 leader_election.cc:302] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02dad6b1d55f410280ba8174338a2c94, a4b6d136b239470eb93776872a9a472c; no voters:
I20260925 02:22:36.213009 14108 raft_consensus.cc:2781] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:36.213148 14108 raft_consensus.cc:686] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [term 1 LEADER]: Becoming Leader. State: Replica: a4b6d136b239470eb93776872a9a472c, State: Running, Role: LEADER
I20260925 02:22:36.213209 14108 consensus_queue.cc:237] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } }
I20260925 02:22:36.213865 13516 catalog_manager.cc:5495] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b6d136b239470eb93776872a9a472c (127.11.125.1). New cstate: current_term: 1 leader_uuid: "a4b6d136b239470eb93776872a9a472c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b6d136b239470eb93776872a9a472c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:36.234969 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:36.232978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.11.125.1:35471)
0925 02:22:36.233030 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:36.233033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:36.233041 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:36.233092 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:36.233093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:36.233104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:36.233147 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:36.233227 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:36.233704 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:36.234395 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:36.234575 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:36.234625 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:36.234726 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:36.234752 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:36.234785 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:36.234839 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:22:36.234855 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:22:36.237557 13817 raft_consensus.cc:1261] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Refusing update from remote peer a4b6d136b239470eb93776872a9a472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:36.237561 13958 raft_consensus.cc:1261] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Refusing update from remote peer a4b6d136b239470eb93776872a9a472c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:36.237797 14152 consensus_queue.cc:1035] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "009ea5b089a3465cbcd0405f182a8a7e" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:36.237938 14108 consensus_queue.cc:1035] T 7173f7bc8f9d4b3ab7b71295fe9341ad P a4b6d136b239470eb93776872a9a472c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02dad6b1d55f410280ba8174338a2c94" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:22:36.242950 14097 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:22:37.761739 13515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:37.761900 13515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:37.762792 13515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 270136b009b84a0aa72de5e0fa4869b7: Sending DeleteTablet for 3 replicas of tablet 7173f7bc8f9d4b3ab7b71295fe9341ad
I20260925 02:22:37.763036 13655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:37 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38434
I20260925 02:22:37.763067 13797 tablet_service.cc:1514] Processing DeleteTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:37 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37562
I20260925 02:22:37.763087 13937 tablet_service.cc:1514] Processing DeleteTablet for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:37 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260925 02:22:37.763268 14182 tablet_replica.cc:330] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: stopping tablet replica
I20260925 02:22:37.763279 14183 tablet_replica.cc:330] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: stopping tablet replica
I20260925 02:22:37.763337 14182 raft_consensus.cc:2229] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:37.763337 14183 raft_consensus.cc:2229] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:37.763383 14183 raft_consensus.cc:2258] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:37.763409 14182 raft_consensus.cc:2258] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:37.763408 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13588
I20260925 02:22:37.764355 14183 ts_tablet_manager.cc:1905] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:37.765364 14182 ts_tablet_manager.cc:1905] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:37.766826 14183 ts_tablet_manager.cc:1918] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:22:37.766932 14183 log.cc:1196] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/7173f7bc8f9d4b3ab7b71295fe9341ad
I20260925 02:22:37.766989 14182 ts_tablet_manager.cc:1918] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:22:37.767066 14182 log.cc:1196] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/7173f7bc8f9d4b3ab7b71295fe9341ad
I20260925 02:22:37.767170 14183 ts_tablet_manager.cc:1939] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 02dad6b1d55f410280ba8174338a2c94: Deleting consensus metadata
I20260925 02:22:37.767357 14182 ts_tablet_manager.cc:1939] T 7173f7bc8f9d4b3ab7b71295fe9341ad P 009ea5b089a3465cbcd0405f182a8a7e: Deleting consensus metadata
I20260925 02:22:37.767544 13501 catalog_manager.cc:4841] TS 02dad6b1d55f410280ba8174338a2c94 (127.11.125.2:38869): tablet 7173f7bc8f9d4b3ab7b71295fe9341ad (table test-table [id=b2a7e2f43b31489d962ba87445db26f8]) successfully deleted
I20260925 02:22:37.767710 13502 catalog_manager.cc:4841] TS 009ea5b089a3465cbcd0405f182a8a7e (127.11.125.3:46123): tablet 7173f7bc8f9d4b3ab7b71295fe9341ad (table test-table [id=b2a7e2f43b31489d962ba87445db26f8]) successfully deleted
W20260925 02:22:37.771179 13501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:35471 (error 108)
I20260925 02:22:37.771456 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13730
I20260925 02:22:37.772629 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:37.771618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:35471 (local address 127.0.0.1:38460)
0925 02:22:37.772517 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:37.772574 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:35471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":62,"threads_started":1}
W20260925 02:22:37.772738 13501 catalog_manager.cc:4584] TS a4b6d136b239470eb93776872a9a472c (127.11.125.1:35471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7173f7bc8f9d4b3ab7b71295fe9341ad: Network error: Client connection negotiation failed: client connection to 127.11.125.1:35471: connect: Connection refused (error 111)
I20260925 02:22:37.777721 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13872
I20260925 02:22:37.784046 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 13484
2026-09-25T02:22:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6509 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:37 dist-test-slave-hj7s krb5kdc[14188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:37 dist-test-slave-hj7s krb5kdc[14188](info): set up 2 sockets
Sep 25 02:22:37 dist-test-slave-hj7s krb5kdc[14188](info): commencing operation
krb5kdc: starting...
W20260925 02:22:38.867121 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:38 dist-test-slave-hj7s krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:38.988536 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44609
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:39.070789 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:39.071030 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:39.071069 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:39.072233 14202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:39.072322 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:39.072350 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:39.072381 14202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:39.072405 14202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:39.073961 14202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:39.074276 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:39.074519 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:39.079919 14211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.079983 14202 server_base.cc:1034] running on GCE node
W20260925 02:22:39.079934 14208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:39.079932 14209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.080404 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:39.080667 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:39.081822 14202 hybrid_clock.cc:648] HybridClock initialized: now 1790302959081803 us; error 37 us; skew 500 ppm
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:39.084267 14202 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:39.084757 14202 webserver.cc:460] Webserver started at http://127.11.125.62:41675/ using document root <none> and password file <none>
I20260925 02:22:39.085031 14202 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:39.085078 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:39.085170 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:39.086194 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "e971bd31bd7e4bac9b415ef4d6f47312"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "54fcc9f9e256c43700596673177265bf"
server_key_iv: "e17aaf1e07498d18e2d2440b2cdac25a"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.086494 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "e971bd31bd7e4bac9b415ef4d6f47312"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "54fcc9f9e256c43700596673177265bf"
server_key_iv: "e17aaf1e07498d18e2d2440b2cdac25a"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.087803 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:39.088534 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.088737 14202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:39.088831 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "e971bd31bd7e4bac9b415ef4d6f47312"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "54fcc9f9e256c43700596673177265bf"
server_key_iv: "e17aaf1e07498d18e2d2440b2cdac25a"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.088893 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:39.105531 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:39.107614 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:39.107735 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:39.111774 14202 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44609
I20260925 02:22:39.111835 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44609 every 8 connection(s)
I20260925 02:22:39.112249 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:39.113303 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:39.113605 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14202
I20260925 02:22:39.113682 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:39.113866 11764 external_mini_cluster.cc:1428] Setting key 7ed6e3d3c87cee1d2a734c593d584f95
I20260925 02:22:39.115887 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: Bootstrap starting.
I20260925 02:22:39.116474 14271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:39.116792 14271 log.cc:824] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:39.117532 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: No bootstrap required, opened a new log
I20260925 02:22:39.119472 14271 raft_consensus.cc:348] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } }
I20260925 02:22:39.119632 14271 raft_consensus.cc:374] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:39.119683 14271 raft_consensus.cc:729] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e971bd31bd7e4bac9b415ef4d6f47312, State: Initialized, Role: FOLLOWER
I20260925 02:22:39.119853 14271 consensus_queue.cc:260] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } }
I20260925 02:22:39.119951 14271 raft_consensus.cc:388] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:39.119988 14271 raft_consensus.cc:482] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:39.120045 14271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:39.120731 14271 raft_consensus.cc:504] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } }
I20260925 02:22:39.120860 14271 leader_election.cc:302] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e971bd31bd7e4bac9b415ef4d6f47312; no voters:
I20260925 02:22:39.121076 14271 leader_election.cc:288] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:39.121155 14276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:39.121321 14276 raft_consensus.cc:686] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [term 1 LEADER]: Becoming Leader. State: Replica: e971bd31bd7e4bac9b415ef4d6f47312, State: Running, Role: LEADER
I20260925 02:22:39.121393 14271 sys_catalog.cc:564] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:39.122103 14276 consensus_queue.cc:237] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } }
I20260925 02:22:39.122921 14278 sys_catalog.cc:455] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } } }
I20260925 02:22:39.123593 14278 sys_catalog.cc:458] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:39.123013 14277 sys_catalog.cc:455] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e971bd31bd7e4bac9b415ef4d6f47312. Latest consensus state: current_term: 1 leader_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e971bd31bd7e4bac9b415ef4d6f47312" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44609 } } }
I20260925 02:22:39.123781 14277 sys_catalog.cc:458] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:39.123946 14291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:39.124010 14291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:39.124095 14294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:39.124310 14294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:39.125135 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.114455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.11.125.62:44609)
0925 02:22:39.114656 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:39.114664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:39.114689 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:22:39.115119 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:39.115126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:39.115173 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:39.115338 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:39.115968 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.117040 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.117867 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.118287 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.118419 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.121174 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:39.121178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:39.121179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:39.121190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:39.123274 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.123713 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.123716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.123717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.123848 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.124075 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.124076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.124076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.124378 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:39.124507 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:39.124774 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:22:39.124945 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260925 02:22:39.126029 14294 catalog_manager.cc:1347] Generated new cluster ID: dae4c8e880cd42338692eca3d162ff47
I20260925 02:22:39.126083 14294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:39.144307 14294 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:39.145105 14294 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:39.151872 14294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e971bd31bd7e4bac9b415ef4d6f47312: Generated new TSK 0
I20260925 02:22:39.152091 14294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:39.184615 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:39.267944 14298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:39.268154 14298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:39.268180 14298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:39.269420 14298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:39.269461 14298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:39.269487 14298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:39.270875 14298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:39.271057 14298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:39.271224 14298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:39.276503 14303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.276657 14298 server_base.cc:1034] running on GCE node
W20260925 02:22:39.276479 14306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:39.276479 14304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.277030 14298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:39.277297 14298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:39.278461 14298 hybrid_clock.cc:648] HybridClock initialized: now 1790302959278449 us; error 28 us; skew 500 ppm
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:39.281111 14298 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:39.281621 14298 webserver.cc:460] Webserver started at http://127.11.125.1:40237/ using document root <none> and password file <none>
I20260925 02:22:39.281855 14298 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:39.281925 14298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:39.282090 14298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:39.283174 14298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "95cb24c29b924e5abb95127669aace5f"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "2c65fc025fb44b2dc22ce61d27e8602b"
server_key_iv: "9d297ea97e0097f23d86a18af6922198"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.283564 14298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "95cb24c29b924e5abb95127669aace5f"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "2c65fc025fb44b2dc22ce61d27e8602b"
server_key_iv: "9d297ea97e0097f23d86a18af6922198"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.285002 14298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:39.285888 14313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.286196 14298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:39.286319 14298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "95cb24c29b924e5abb95127669aace5f"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "2c65fc025fb44b2dc22ce61d27e8602b"
server_key_iv: "9d297ea97e0097f23d86a18af6922198"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.286396 14298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:39.297009 14298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:39.299280 14298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:39.299439 14298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:39.299669 14298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:39.299978 14298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:39.300031 14298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.300073 14298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:39.300101 14298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.306221 14298 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:42187
I20260925 02:22:39.306269 14428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:42187 every 8 connection(s)
I20260925 02:22:39.306583 14298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:39.309742 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14298
I20260925 02:22:39.309855 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:39.310045 11764 external_mini_cluster.cc:1428] Setting key 064fd628759e6107e806cc370dc24a01
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:39.316635 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.307332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33821 (local address 127.11.125.62:44609)
0925 02:22:39.307411 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:39.307413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:39.307661 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:22:39.308159 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:39.308160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:39.308170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:39.308229 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:39.309276 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.309794 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.311185 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.311423 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.311514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.313806 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:39.313808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:39.313808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:39.313816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:39.315206 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.315634 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.315635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.315636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.315700 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.315969 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.315970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.315970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.316088 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:39.316167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:39.316449 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:22:39.316543 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:22:39.316831 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.307312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44609 (local address 127.11.125.1:33821)
0925 02:22:39.307507 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:39.307559 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:39.308071 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:39.308301 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:39.308305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:39.308987 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:39.309183 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.309189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.309905 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.309906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.311093 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.311096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.311527 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.311528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.311662 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.312179 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:39.312188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:39.313697 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:39.315321 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.315329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.315332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.315556 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.315784 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.315785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.315786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.315900 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.316157 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:39.316165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:39.316310 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:22:39.316415 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:22:39.316585 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260925 02:22:39.317533 14429 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44609
I20260925 02:22:39.317618 14429 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:39.317838 14429 heartbeater.cc:511] Master 127.11.125.62:44609 requested a full tablet report, sending...
I20260925 02:22:39.318451 14235 ts_manager.cc:194] Registered new tserver with Master: 95cb24c29b924e5abb95127669aace5f (127.11.125.1:42187)
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:22:39.319352 14235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:33821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:39.335871 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:39.417665 14434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:39.417872 14434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:39.417896 14434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:39.419029 14434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:39.419068 14434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:39.419094 14434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:39.420537 14434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:39.420723 14434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:39.420946 14434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:39.426131 14442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:39.426134 14440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.426235 14434 server_base.cc:1034] running on GCE node
W20260925 02:22:39.426134 14439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.426599 14434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:39.426859 14434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:39.428030 14434 hybrid_clock.cc:648] HybridClock initialized: now 1790302959428023 us; error 37 us; skew 500 ppm
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:39.430562 14434 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:39.431016 14434 webserver.cc:460] Webserver started at http://127.11.125.2:41723/ using document root <none> and password file <none>
I20260925 02:22:39.431214 14434 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:39.431253 14434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:39.431370 14434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:39.432399 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "338e359daf7247d2af66775dc6e580d6"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "e3e6fbabf0c7b78f7af493ec79ea0a6c"
server_key_iv: "145a252d8e201728f50a9a23014eccc5"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.432700 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "338e359daf7247d2af66775dc6e580d6"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "e3e6fbabf0c7b78f7af493ec79ea0a6c"
server_key_iv: "145a252d8e201728f50a9a23014eccc5"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.434065 14434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:39.434770 14449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.434934 14434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:39.435019 14434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "338e359daf7247d2af66775dc6e580d6"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "e3e6fbabf0c7b78f7af493ec79ea0a6c"
server_key_iv: "145a252d8e201728f50a9a23014eccc5"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.435074 14434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:39.452380 14434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:39.454756 14434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:39.454872 14434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:39.455076 14434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:39.455402 14434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:39.455437 14434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.455461 14434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:39.455508 14434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.461406 14434 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:39809
I20260925 02:22:39.461598 14563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:39809 every 8 connection(s)
I20260925 02:22:39.461830 14434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:39.470698 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14434
I20260925 02:22:39.470808 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:39.470842 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.462584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:34555 (local address 127.11.125.62:44609)
0925 02:22:39.462669 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:39.462672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:39.462844 (+ 172us) server_negotiation.cc:407] Connection header received
0925 02:22:39.463345 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:39.463346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:39.463356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:39.463409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:39.464342 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.464852 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.465890 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.466068 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.466121 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.468192 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:39.468194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:39.468194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:39.468200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:39.469608 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.469964 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.469965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.469966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.470016 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.470222 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.470223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.470223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.470332 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:39.470382 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:39.470645 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:22:39.470731 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:22:39.471005 11764 external_mini_cluster.cc:1428] Setting key c9ccd181daed9da550deb9c653c02046
I20260925 02:22:39.470993 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.462525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44609 (local address 127.11.125.2:34555)
0925 02:22:39.462686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:39.462723 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:39.463250 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:39.463466 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:39.463469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:39.464044 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:39.464250 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.464256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.464987 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.464988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.465806 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.465808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.466144 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.466145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.466255 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.466736 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:39.466744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:39.468104 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:39.469677 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.469684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.469685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.469901 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.470073 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.470073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.470074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.470171 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.470386 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:39.470391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:39.470519 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 02:22:39.470599 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:22:39.470717 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260925 02:22:39.471588 14564 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44609
I20260925 02:22:39.471695 14564 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:39.471886 14564 heartbeater.cc:511] Master 127.11.125.62:44609 requested a full tablet report, sending...
I20260925 02:22:39.472317 14234 ts_manager.cc:194] Registered new tserver with Master: 338e359daf7247d2af66775dc6e580d6 (127.11.125.2:39809)
I20260925 02:22:39.472721 14234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:34555
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:39.496338 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:39.577039 14569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:39.577235 14569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:39.577266 14569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:39.578347 14569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:39.578380 14569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:39.578406 14569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:39.579747 14569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:39.579919 14569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:39.580081 14569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:22:39.585237 14569 server_base.cc:1034] running on GCE node
W20260925 02:22:39.585220 14574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:39.585244 14575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:39.585210 14577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:39.585673 14569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:39.585930 14569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:39.587108 14569 hybrid_clock.cc:648] HybridClock initialized: now 1790302959587093 us; error 27 us; skew 500 ppm
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:39.597684 14569 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:39.598151 14569 webserver.cc:460] Webserver started at http://127.11.125.3:45021/ using document root <none> and password file <none>
I20260925 02:22:39.598395 14569 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:39.598465 14569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:39.598624 14569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:39.599735 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "119a5adc1e4f44a5b4fbee79aa0a722b"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "53e88e383bba3353a98e4beeb33f9125"
server_key_iv: "f4e5921d52dc10873f99df3beb8d51ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.600103 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "119a5adc1e4f44a5b4fbee79aa0a722b"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "53e88e383bba3353a98e4beeb33f9125"
server_key_iv: "f4e5921d52dc10873f99df3beb8d51ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.601522 14569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:39.602262 14584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.602465 14569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:39.602578 14569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "119a5adc1e4f44a5b4fbee79aa0a722b"
format_stamp: "Formatted at 2026-09-25 02:22:39 on dist-test-slave-hj7s"
server_key: "53e88e383bba3353a98e4beeb33f9125"
server_key_iv: "f4e5921d52dc10873f99df3beb8d51ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:39.602679 14569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:39.613819 14569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:39.616122 14569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:39.616299 14569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:39.616532 14569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:39.616849 14569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:39.616905 14569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.616979 14569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:39.617010 14569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:39.623245 14569 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:40455
I20260925 02:22:39.623277 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:40455 every 8 connection(s)
I20260925 02:22:39.623639 14569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302959, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:39.631222 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14569
I20260925 02:22:39.631333 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:39.631542 11764 external_mini_cluster.cc:1428] Setting key 79c2a4121190197983a461c49915bb0f
I20260925 02:22:39.633773 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.624380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:39091 (local address 127.11.125.62:44609)
0925 02:22:39.624424 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:39.624426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:39.624612 (+ 186us) server_negotiation.cc:407] Connection header received
0925 02:22:39.625221 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:39.625223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:39.625236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:39.625303 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:39.626234 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.626788 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.627796 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.628012 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.628098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.630546 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:39.630548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:39.630548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:39.630557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:39.632349 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.632747 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.632748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.632749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.632806 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:39.633102 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:39.633103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:39.633104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:39.633236 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:39.633316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:39.633607 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:22:39.633685 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:22:39.633958 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:39.624345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44609 (local address 127.11.125.3:39091)
0925 02:22:39.624486 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:39.624529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:39.625116 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:39.625379 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:39.625382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:39.625943 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:39.626132 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.626137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.626889 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.626890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.627681 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:39.627684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:39.628107 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:39.628108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:39.628234 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:39.628775 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:39.628784 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:39.630444 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:39.632409 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.632417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.632419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.632651 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.632882 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:39.632883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:39.632883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:39.633029 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:39.633306 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:39.633314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:39.633457 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 02:22:39.633568 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:22:39.633724 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260925 02:22:39.634383 14699 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44609
I20260925 02:22:39.634495 14699 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:39.634675 14699 heartbeater.cc:511] Master 127.11.125.62:44609 requested a full tablet report, sending...
I20260925 02:22:39.635039 14234 ts_manager.cc:194] Registered new tserver with Master: 119a5adc1e4f44a5b4fbee79aa0a722b (127.11.125.3:40455)
I20260925 02:22:39.635424 14234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:39091
I20260925 02:22:39.643375 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:22:39.655280 14274 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:39.654325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.11.125.62:44609)
0925 02:22:39.654393 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:39.654395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:39.654516 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:22:39.654563 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:39.654564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:39.654573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:39.654624 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:39.655197 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37462: BlockingRecv error: recv got EOF from 127.0.0.1:37462 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:22:39.656121 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14298
I20260925 02:22:39.661347 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14434
I20260925 02:22:39.666010 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14569
I20260925 02:22:39.670604 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14202
2026-09-25T02:22:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1884 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14711](info): set up 2 sockets
Sep 25 02:22:39 dist-test-slave-hj7s krb5kdc[14711](info): commencing operation
krb5kdc: starting...
W20260925 02:22:40.729029 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:40 dist-test-slave-hj7s krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:40.847800 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35187
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:40.929733 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:40.929983 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:40.930022 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:40.931231 14724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:40.931322 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:40.931349 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:40.931381 14724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:40.931404 14724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:40.932981 14724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:40.933266 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:40.933491 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:40.938441 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:40.938436 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:40.938441 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:40.938763 14724 server_base.cc:1034] running on GCE node
I20260925 02:22:40.939016 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:40.939311 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:40.940474 14724 hybrid_clock.cc:648] HybridClock initialized: now 1790302960940460 us; error 44 us; skew 500 ppm
Sep 25 02:22:40 dist-test-slave-hj7s krb5kdc[14711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302960, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:40.943099 14724 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:40.943552 14724 webserver.cc:460] Webserver started at http://127.11.125.62:36999/ using document root <none> and password file <none>
I20260925 02:22:40.943781 14724 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:40.943850 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:40.944007 14724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:40.945205 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "697ae30d10594080a7652a3fc7bdf0a7"
format_stamp: "Formatted at 2026-09-25 02:22:40 on dist-test-slave-hj7s"
server_key: "cce660a9da1ba67a1132426bd8c02b88"
server_key_iv: "7495baea717c814d267bd3fbf79fbab3"
server_key_version: "encryptionkey@0"
I20260925 02:22:40.945576 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "697ae30d10594080a7652a3fc7bdf0a7"
format_stamp: "Formatted at 2026-09-25 02:22:40 on dist-test-slave-hj7s"
server_key: "cce660a9da1ba67a1132426bd8c02b88"
server_key_iv: "7495baea717c814d267bd3fbf79fbab3"
server_key_version: "encryptionkey@0"
I20260925 02:22:40.946902 14724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:40.947695 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:40.947903 14724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:40.948014 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "697ae30d10594080a7652a3fc7bdf0a7"
format_stamp: "Formatted at 2026-09-25 02:22:40 on dist-test-slave-hj7s"
server_key: "cce660a9da1ba67a1132426bd8c02b88"
server_key_iv: "7495baea717c814d267bd3fbf79fbab3"
server_key_version: "encryptionkey@0"
I20260925 02:22:40.948089 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:40.966895 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:40.969425 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:40.969611 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:40.973817 14724 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:35187
I20260925 02:22:40.973862 14792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:35187 every 8 connection(s)
I20260925 02:22:40.974404 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:40.975135 14793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:40.977523 14793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7: Bootstrap starting.
I20260925 02:22:40.978155 14793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:40.978471 14793 log.cc:824] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:40.979171 14793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7: No bootstrap required, opened a new log
I20260925 02:22:40.980782 14793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } }
I20260925 02:22:40.981006 14793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:40.981071 14793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697ae30d10594080a7652a3fc7bdf0a7, State: Initialized, Role: FOLLOWER
I20260925 02:22:40.981225 14793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } }
I20260925 02:22:40.981298 14793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:40.981321 14793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:40.981393 14793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:40.982317 14793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } }
I20260925 02:22:40.982409 14793 leader_election.cc:302] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 697ae30d10594080a7652a3fc7bdf0a7; no voters:
I20260925 02:22:40.982628 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14724
I20260925 02:22:40.982635 14793 leader_election.cc:288] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:40.982669 14796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:40.982738 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:40.982793 14796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 697ae30d10594080a7652a3fc7bdf0a7, State: Running, Role: LEADER
I20260925 02:22:40.982940 11764 external_mini_cluster.cc:1428] Setting key e6cc4a83f0318c503b186841f2ea01a2
I20260925 02:22:40.982932 14793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:40.982946 14796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } }
I20260925 02:22:40.983282 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "697ae30d10594080a7652a3fc7bdf0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } } }
I20260925 02:22:40.983374 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:40.983289 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697ae30d10594080a7652a3fc7bdf0a7. Latest consensus state: current_term: 1 leader_uuid: "697ae30d10594080a7652a3fc7bdf0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697ae30d10594080a7652a3fc7bdf0a7" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35187 } } }
I20260925 02:22:40.983750 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:40.983946 14803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:40.984599 14803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:40.986366 14803 catalog_manager.cc:1347] Generated new cluster ID: 5306c8293c7e4e67b1acc0d74530c327
I20260925 02:22:40.986416 14803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:22:40 dist-test-slave-hj7s krb5kdc[14711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:40.994781 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:40.983453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.11.125.62:35187)
0925 02:22:40.983649 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:40.983660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:40.983786 (+ 126us) server_negotiation.cc:407] Connection header received
0925 02:22:40.983919 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:40.983927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:40.983990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:40.984133 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:40.985084 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:40.986438 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:40.987487 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:40.988153 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:40.988274 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:40.991081 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:40.991085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:40.991085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:40.991097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:40.993139 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:40.993539 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:40.993543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:40.993543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:40.993651 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:40.993877 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:40.993878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:40.993878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:40.994165 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:40.994326 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:40.994428 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:22:40.994564 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260925 02:22:40.995882 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:22:40.999925 14803 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:41.000598 14803 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:41.005213 14803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 697ae30d10594080a7652a3fc7bdf0a7: Generated new TSK 0
I20260925 02:22:41.005394 14803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:41.011731 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:41.007991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.11.125.62:35187)
0925 02:22:41.008030 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:41.008032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:41.008205 (+ 173us) server_negotiation.cc:407] Connection header received
0925 02:22:41.008295 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:41.008297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:41.008316 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:41.008387 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:41.008808 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:41.009408 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:41.010200 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:41.010433 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:41.010511 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:41.011239 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:41.011241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:41.011241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:41.011451 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:41.011467 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:41.011473 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:41.011544 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:41.011611 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:22:41.011636 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:22:41.014245 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14724
2026-09-25T02:22:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1340 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:41 dist-test-slave-hj7s krb5kdc[14826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:41 dist-test-slave-hj7s krb5kdc[14826](info): set up 2 sockets
Sep 25 02:22:41 dist-test-slave-hj7s krb5kdc[14826](info): commencing operation
krb5kdc: starting...
W20260925 02:22:42.063651 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:42.184608 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38113
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:38113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:42.265777 14839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:42.266007 14839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:42.266042 14839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:42.267283 14839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:42.267359 14839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:42.267383 14839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:42.267402 14839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:42.267434 14839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:42.268977 14839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:38113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:42.269258 14839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:42.269495 14839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:42.274610 14845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.274586 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.274598 14844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.274644 14839 server_base.cc:1034] running on GCE node
I20260925 02:22:42.275101 14839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:42.275390 14839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:42.276558 14839 hybrid_clock.cc:648] HybridClock initialized: now 1790302962276553 us; error 38 us; skew 500 ppm
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:42.279147 14839 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:42.279536 14839 webserver.cc:460] Webserver started at http://127.11.125.62:39719/ using document root <none> and password file <none>
I20260925 02:22:42.279734 14839 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:42.279778 14839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:42.279865 14839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:42.280879 14839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "d7f90452c0204d4a9fbb6971f5cbc376"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "84c8fe707e76a7f971e4f68b6313e64c"
server_key_iv: "23542208019bc17d1a9ccf9386c99376"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.281262 14839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "d7f90452c0204d4a9fbb6971f5cbc376"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "84c8fe707e76a7f971e4f68b6313e64c"
server_key_iv: "23542208019bc17d1a9ccf9386c99376"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.282543 14839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:42.283538 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.283723 14839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:42.283834 14839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "d7f90452c0204d4a9fbb6971f5cbc376"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "84c8fe707e76a7f971e4f68b6313e64c"
server_key_iv: "23542208019bc17d1a9ccf9386c99376"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.283923 14839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:42.301321 14839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:42.303174 14839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:42.303354 14839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:42.307554 14839 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:38113
I20260925 02:22:42.307626 14906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:38113 every 8 connection(s)
I20260925 02:22:42.307999 14839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:42.309072 14907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:42.309481 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14839
I20260925 02:22:42.309609 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:42.309800 11764 external_mini_cluster.cc:1428] Setting key aee2d45a545c8dd35bcedca14939cc66
I20260925 02:22:42.312323 14907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: Bootstrap starting.
I20260925 02:22:42.313004 14907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:42.313323 14907 log.cc:824] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:42.314007 14907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: No bootstrap required, opened a new log
I20260925 02:22:42.316043 14907 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } }
I20260925 02:22:42.316210 14907 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:42.316263 14907 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f90452c0204d4a9fbb6971f5cbc376, State: Initialized, Role: FOLLOWER
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:42.316452 14907 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } }
I20260925 02:22:42.316561 14907 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:42.316597 14907 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:42.316639 14907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:42.317394 14907 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } }
I20260925 02:22:42.317514 14907 leader_election.cc:302] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7f90452c0204d4a9fbb6971f5cbc376; no voters:
I20260925 02:22:42.317691 14907 leader_election.cc:288] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:42.317747 14912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:42.317879 14912 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [term 1 LEADER]: Becoming Leader. State: Replica: d7f90452c0204d4a9fbb6971f5cbc376, State: Running, Role: LEADER
I20260925 02:22:42.317988 14907 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:42.318257 14912 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } }
I20260925 02:22:42.319115 14914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } } }
I20260925 02:22:42.319212 14914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:42.319687 14913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f90452c0204d4a9fbb6971f5cbc376. Latest consensus state: current_term: 1 leader_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f90452c0204d4a9fbb6971f5cbc376" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38113 } } }
I20260925 02:22:42.320098 14913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:42.320218 14928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:42.320292 14928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:42.320434 14923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:42.320618 14923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:42.321295 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.310378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.11.125.62:38113)
0925 02:22:42.310569 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:42.310585 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:42.310610 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:22:42.311036 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:42.311042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:42.311087 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:42.311202 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:42.312025 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.313025 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.313924 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.314391 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.314504 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.317080 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:42.317084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:42.317084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:42.317095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:42.319146 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.319584 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.319588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.319588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.319698 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.319939 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.319940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.319941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.320266 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:42.320414 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:42.320943 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 02:22:42.321081 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260925 02:22:42.322330 14923 catalog_manager.cc:1347] Generated new cluster ID: e6f42b758ecc4bb8939496ca407fa13b
I20260925 02:22:42.322386 14923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:42.346293 14923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:42.347043 14923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:42.357055 14923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7f90452c0204d4a9fbb6971f5cbc376: Generated new TSK 0
I20260925 02:22:42.357268 14923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:42.384836 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:42.470211 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:42.470436 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:42.470461 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:42.471674 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:42.471719 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:42.471746 14933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:42.473441 14933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:42.473706 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:42.473948 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:42.479172 14939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.479221 14938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.479249 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.479481 14933 server_base.cc:1034] running on GCE node
I20260925 02:22:42.479707 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:42.479974 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:42.481117 14933 hybrid_clock.cc:648] HybridClock initialized: now 1790302962481104 us; error 29 us; skew 500 ppm
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:42.483712 14933 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:42.484247 14933 webserver.cc:460] Webserver started at http://127.11.125.1:44887/ using document root <none> and password file <none>
I20260925 02:22:42.484506 14933 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:42.484573 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:42.484768 14933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:42.486003 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "3348e8e3834544569d4d18fff4e302f3"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "d1450ec1a82eed7df1040514d75461f3"
server_key_iv: "9b2a752ab2eddfbc8de5fbd024792bc9"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.486449 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "3348e8e3834544569d4d18fff4e302f3"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "d1450ec1a82eed7df1040514d75461f3"
server_key_iv: "9b2a752ab2eddfbc8de5fbd024792bc9"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.488077 14933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:42.488941 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.489142 14933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:42.489231 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "3348e8e3834544569d4d18fff4e302f3"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "d1450ec1a82eed7df1040514d75461f3"
server_key_iv: "9b2a752ab2eddfbc8de5fbd024792bc9"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.489293 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:42.507846 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:42.510070 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:42.510195 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:42.510421 14933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:42.510782 14933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:42.510824 14933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.510848 14933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:42.510885 14933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.516937 14933 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:42061
I20260925 02:22:42.517010 15063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:42061 every 8 connection(s)
I20260925 02:22:42.517351 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:42.520236 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 14933
I20260925 02:22:42.520383 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:42.520598 11764 external_mini_cluster.cc:1428] Setting key fb6f24eb8204c757db2e2f3efd7e4bd9
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:42.528515 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.518166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:49383 (local address 127.11.125.62:38113)
0925 02:22:42.518231 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:42.518234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:42.518472 (+ 238us) server_negotiation.cc:407] Connection header received
0925 02:22:42.519133 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:42.519134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:42.519147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:42.519244 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:42.520322 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.521112 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.522214 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.522561 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.522639 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.525167 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:42.525169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:42.525169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:42.525178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:42.527001 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.527449 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.527451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.527452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.527521 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.527800 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.527801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.527802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.527948 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:42.528028 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:42.528304 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:22:42.528398 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:42.528553 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.518135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38113 (local address 127.11.125.1:49383)
0925 02:22:42.518331 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:42.518383 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:42.519013 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:42.519341 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:42.519345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:42.520004 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:42.520215 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.520221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.521249 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.521251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.522113 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.522115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.522662 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.522663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.522820 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.523394 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:42.523405 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:42.525075 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:42.527129 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.527136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.527138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.527364 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.527597 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.527598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.527598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.527714 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.528020 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:42.528025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:42.528167 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 02:22:42.528254 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:22:42.528379 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260925 02:22:42.529175 15064 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38113
I20260925 02:22:42.529282 15064 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:22:42.529516 15064 heartbeater.cc:511] Master 127.11.125.62:38113 requested a full tablet report, sending...
I20260925 02:22:42.530117 14871 ts_manager.cc:194] Registered new tserver with Master: 3348e8e3834544569d4d18fff4e302f3 (127.11.125.1:42061)
I20260925 02:22:42.530856 14871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:49383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:42.547811 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:42.630808 15069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:42.631012 15069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:42.631036 15069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:42.632225 15069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:42.632269 15069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:42.632296 15069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:42.633831 15069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:42.634025 15069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:42.634215 15069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:42.639315 15075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.639273 15074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.639385 15069 server_base.cc:1034] running on GCE node
W20260925 02:22:42.639338 15077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.639724 15069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:42.639979 15069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:42.641140 15069 hybrid_clock.cc:648] HybridClock initialized: now 1790302962641127 us; error 43 us; skew 500 ppm
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:42.643735 15069 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:42.644124 15069 webserver.cc:460] Webserver started at http://127.11.125.2:40847/ using document root <none> and password file <none>
I20260925 02:22:42.644322 15069 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:42.644363 15069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:42.644449 15069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:42.645478 15069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "41ac1cdb671e4580b677ba84c1fbd278"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "88254b125250843289ac3396c3ce2a10"
server_key_iv: "2828b15e3a53def41a25a47e73ddf0a6"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.645777 15069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "41ac1cdb671e4580b677ba84c1fbd278"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "88254b125250843289ac3396c3ce2a10"
server_key_iv: "2828b15e3a53def41a25a47e73ddf0a6"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.646996 15069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:42.647671 15084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.647848 15069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:42.647928 15069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "41ac1cdb671e4580b677ba84c1fbd278"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "88254b125250843289ac3396c3ce2a10"
server_key_iv: "2828b15e3a53def41a25a47e73ddf0a6"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.647980 15069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:42.661083 15069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:42.662937 15069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:42.663051 15069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:42.663236 15069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:42.663501 15069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:42.663530 15069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.663552 15069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:42.663565 15069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.669284 15069 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:41395
I20260925 02:22:42.669374 15198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:41395 every 8 connection(s)
I20260925 02:22:42.669682 15069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:42.672832 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15069
I20260925 02:22:42.672974 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:42.673187 11764 external_mini_cluster.cc:1428] Setting key a20f6138787aae18a38619bce9e4003a
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:42.680639 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.670428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:54751 (local address 127.11.125.62:38113)
0925 02:22:42.670509 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:42.670511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:42.670687 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:22:42.671240 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:42.671242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:42.671254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:42.671338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:42.672298 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.673017 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.674394 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.674719 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.674796 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.677650 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:42.677651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:42.677652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:42.677659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:42.679078 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.679637 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.679638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.679638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.679689 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.679985 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.679986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.679986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.680088 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:42.680139 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:42.680505 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 02:22:42.680572 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:22:42.680963 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.670375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38113 (local address 127.11.125.2:54751)
0925 02:22:42.670544 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:42.670595 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:42.671125 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:42.671413 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:42.671417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:42.671994 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:42.672183 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.672188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.673124 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.673125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.674299 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.674302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.674808 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.674809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.674913 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.675592 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:42.675602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:42.677555 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:42.679168 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.679177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.679179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.679550 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.679757 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.679758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.679759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.679917 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.680150 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:42.680167 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:42.680360 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 02:22:42.680486 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:22:42.680672 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260925 02:22:42.681551 15199 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38113
I20260925 02:22:42.681670 15199 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:42.681870 15199 heartbeater.cc:511] Master 127.11.125.62:38113 requested a full tablet report, sending...
I20260925 02:22:42.682323 14871 ts_manager.cc:194] Registered new tserver with Master: 41ac1cdb671e4580b677ba84c1fbd278 (127.11.125.2:41395)
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:22:42.682832 14871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:54751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:42.700008 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:42.782737 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:42.782987 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:42.783022 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:42.784201 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:42.784267 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:42.784312 15205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:42.785804 15205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:42.786072 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:42.786335 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:42.791460 15213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:42.791489 15211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.791637 15205 server_base.cc:1034] running on GCE node
W20260925 02:22:42.791460 15210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:42.791922 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:42.792235 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:42.793396 15205 hybrid_clock.cc:648] HybridClock initialized: now 1790302962793384 us; error 32 us; skew 500 ppm
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:42.795805 15205 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:42.796290 15205 webserver.cc:460] Webserver started at http://127.11.125.3:34281/ using document root <none> and password file <none>
I20260925 02:22:42.796519 15205 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:42.796589 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:42.796744 15205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:42.797955 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "ec9120a9011d419b9eb0ff308046dff8"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "b57c2ab2b2aca0ce0a70d6291f6050aa"
server_key_iv: "f426aef1948b521509139c0519b626df"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.798362 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "ec9120a9011d419b9eb0ff308046dff8"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "b57c2ab2b2aca0ce0a70d6291f6050aa"
server_key_iv: "f426aef1948b521509139c0519b626df"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.799718 15205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:42.800444 15220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.800621 15205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:42.800742 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "ec9120a9011d419b9eb0ff308046dff8"
format_stamp: "Formatted at 2026-09-25 02:22:42 on dist-test-slave-hj7s"
server_key: "b57c2ab2b2aca0ce0a70d6291f6050aa"
server_key_iv: "f426aef1948b521509139c0519b626df"
server_key_version: "encryptionkey@0"
I20260925 02:22:42.800822 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:42.822837 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:42.824663 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:42.824834 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:42.825098 15205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:42.825430 15205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:42.825486 15205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.825531 15205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:42.825555 15205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:42.831610 15205 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:38459
I20260925 02:22:42.831660 15333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:38459 every 8 connection(s)
I20260925 02:22:42.831998 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:42.835150 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15205
I20260925 02:22:42.835268 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:42.835453 11764 external_mini_cluster.cc:1428] Setting key 9f56009898868ae4205afc03354a7a80
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:42.841830 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.832722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49293 (local address 127.11.125.62:38113)
0925 02:22:42.832807 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:42.832809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:42.833037 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:22:42.833537 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:42.833538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:42.833548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:42.833617 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:42.834530 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.835028 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.836486 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.836678 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.836734 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.838901 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:42.838903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:42.838903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:42.838911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:42.840578 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.840998 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.840999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.840999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.841055 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.841288 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.841289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.841289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.841396 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:42.841444 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:42.841690 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:22:42.841756 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:22:42.841965 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.832712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38113 (local address 127.11.125.3:49293)
0925 02:22:42.832877 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:42.832944 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:42.833452 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:42.833675 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:42.833678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:42.834257 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:42.834453 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.834458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.835125 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.835126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.836413 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:42.836416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.836758 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.836759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:42.836860 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.837401 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:42.837410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:42.838787 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:42.840651 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.840657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.840658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.840869 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.841119 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:42.841120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:42.841120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:42.841221 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:42.841459 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:42.841464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:42.841575 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:22:42.841651 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:22:42.841769 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260925 02:22:42.842433 15334 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38113
I20260925 02:22:42.842517 15334 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:42.842656 15334 heartbeater.cc:511] Master 127.11.125.62:38113 requested a full tablet report, sending...
I20260925 02:22:42.842973 14871 ts_manager.cc:194] Registered new tserver with Master: ec9120a9011d419b9eb0ff308046dff8 (127.11.125.3:38459)
I20260925 02:22:42.843343 14871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:49293
I20260925 02:22:42.847021 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[14826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302962, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:42.875101 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:42.862220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.11.125.62:38113)
0925 02:22:42.862313 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:42.862316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:42.862381 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:22:42.862428 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:42.862429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:42.862438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:42.862488 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:42.863026 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.863500 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.864183 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:42.864383 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:42.864433 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:42.872658 (+ 8225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:42.872660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:42.872660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:42.872668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:42.874097 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.874405 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.874406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.874406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.874457 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:42.874618 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:42.874619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:42.874619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:42.874781 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:42.874859 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:42.874943 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:22:42.875004 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 02:22:42.875393 14871 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58740
I20260925 02:22:42.876124 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14933
I20260925 02:22:42.882081 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15069
I20260925 02:22:42.887761 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15205
I20260925 02:22:42.892661 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 14839
2026-09-25T02:22:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1880 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[15347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[15347](info): set up 2 sockets
Sep 25 02:22:42 dist-test-slave-hj7s krb5kdc[15347](info): commencing operation
krb5kdc: starting...
W20260925 02:22:43.959174 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:44.079914 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:45071
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:44.161298 15360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:44.161504 15360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:44.161526 15360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:44.162768 15360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:44.162812 15360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:44.162827 15360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:44.162839 15360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:44.162851 15360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:44.164291 15360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:44.164487 15360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:44.164665 15360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:22:44.169570 15360 server_base.cc:1034] running on GCE node
W20260925 02:22:44.169512 15366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.169517 15368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.169540 15365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:44.169986 15360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:44.170323 15360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:44.171494 15360 hybrid_clock.cc:648] HybridClock initialized: now 1790302964171477 us; error 28 us; skew 500 ppm
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:44.174054 15360 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:44.174538 15360 webserver.cc:460] Webserver started at http://127.11.125.62:33617/ using document root <none> and password file <none>
I20260925 02:22:44.174783 15360 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:44.174853 15360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:44.175009 15360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:44.176111 15360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "34554561392848dd9881c52780f917cf"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "89ffabc60cbe6603cc421bbc75b7da75"
server_key_iv: "96cd611d904acdbf4d93fb4037c051ff"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.176515 15360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "34554561392848dd9881c52780f917cf"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "89ffabc60cbe6603cc421bbc75b7da75"
server_key_iv: "96cd611d904acdbf4d93fb4037c051ff"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.177942 15360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:44.178651 15375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.178828 15360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:44.178939 15360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "34554561392848dd9881c52780f917cf"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "89ffabc60cbe6603cc421bbc75b7da75"
server_key_iv: "96cd611d904acdbf4d93fb4037c051ff"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.179016 15360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:44.189392 15360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:44.191175 15360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:44.191354 15360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:44.195488 15360 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:45071
I20260925 02:22:44.195524 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:45071 every 8 connection(s)
I20260925 02:22:44.196038 15360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:44.196745 15428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:44.199040 15428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf: Bootstrap starting.
I20260925 02:22:44.199590 15428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:44.199826 15428 log.cc:824] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:44.200471 15428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf: No bootstrap required, opened a new log
I20260925 02:22:44.202005 15428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } }
I20260925 02:22:44.202144 15428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:44.202167 15428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34554561392848dd9881c52780f917cf, State: Initialized, Role: FOLLOWER
I20260925 02:22:44.202309 15428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } }
I20260925 02:22:44.202395 15428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:44.202418 15428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:44.202447 15428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:44.203310 15428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } }
I20260925 02:22:44.203401 15428 leader_election.cc:302] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34554561392848dd9881c52780f917cf; no voters:
I20260925 02:22:44.203539 15428 leader_election.cc:288] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:44.203621 15431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:44.203821 15431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [term 1 LEADER]: Becoming Leader. State: Replica: 34554561392848dd9881c52780f917cf, State: Running, Role: LEADER
I20260925 02:22:44.203826 15428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:44.204003 15431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } }
I20260925 02:22:44.204412 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34554561392848dd9881c52780f917cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } } }
I20260925 02:22:44.204401 15433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34554561392848dd9881c52780f917cf. Latest consensus state: current_term: 1 leader_uuid: "34554561392848dd9881c52780f917cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34554561392848dd9881c52780f917cf" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 45071 } } }
I20260925 02:22:44.204512 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:44.204519 15433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:44.204816 15440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:44.204936 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15360
I20260925 02:22:44.205035 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:44.205228 11764 external_mini_cluster.cc:1428] Setting key a3d581ec26944c29e66831965f9df05f
I20260925 02:22:44.205394 15440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:44.208913 15440 catalog_manager.cc:1347] Generated new cluster ID: 3ec73795520b4a5bb7fabae3698f440d
I20260925 02:22:44.208987 15440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:44.216095 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.205807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.11.125.62:45071)
0925 02:22:44.205980 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.205989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.206028 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:22:44.206144 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.206151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.206208 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.206371 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.207191 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.208117 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.208852 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.209491 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.209595 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.212410 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.212414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.212414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.212427 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.214479 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.214865 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.214869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.214869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.214979 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.215182 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.215183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.215184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.215507 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.215660 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.215735 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.215880 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260925 02:22:44.218986 15440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:44.219655 15440 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:44.224287 15440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34554561392848dd9881c52780f917cf: Generated new TSK 0
I20260925 02:22:44.224455 15440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:44.254187 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:44.337054 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:44.337306 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:44.337345 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:44.338541 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:44.338619 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:44.338665 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:44.340206 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:44.340474 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:44.340767 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:44.345893 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.345939 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.345913 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:44.346176 15454 server_base.cc:1034] running on GCE node
I20260925 02:22:44.346421 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:44.346671 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:44.347839 15454 hybrid_clock.cc:648] HybridClock initialized: now 1790302964347835 us; error 39 us; skew 500 ppm
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:44.350414 15454 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:44.350859 15454 webserver.cc:460] Webserver started at http://127.11.125.1:34961/ using document root <none> and password file <none>
I20260925 02:22:44.351087 15454 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:44.351155 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:44.351329 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:44.352401 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "531cdc1726f5430fa39ff8417f6992bd"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "2b98c9b82d3a799dbd1c7febcbabe459"
server_key_iv: "03d2066263821b79ffd84ceea3f078ec"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.352787 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "531cdc1726f5430fa39ff8417f6992bd"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "2b98c9b82d3a799dbd1c7febcbabe459"
server_key_iv: "03d2066263821b79ffd84ceea3f078ec"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.354159 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:44.354907 15469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.355086 15454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.355189 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "531cdc1726f5430fa39ff8417f6992bd"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "2b98c9b82d3a799dbd1c7febcbabe459"
server_key_iv: "03d2066263821b79ffd84ceea3f078ec"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.355278 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:44.373981 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:44.376066 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:44.376259 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:44.376528 15454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:44.376875 15454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:44.376966 15454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.377005 15454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:44.377034 15454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.383428 15454 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:40987
I20260925 02:22:44.383477 15583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:40987 every 8 connection(s)
I20260925 02:22:44.383810 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:44.389648 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15454
I20260925 02:22:44.389735 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:44.389941 11764 external_mini_cluster.cc:1428] Setting key 01b2e392071053b7973655c1e181ce73
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:44.393467 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.384616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:58819 (local address 127.11.125.62:45071)
0925 02:22:44.384669 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.384672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.384865 (+ 193us) server_negotiation.cc:407] Connection header received
0925 02:22:44.385864 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.385866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.385880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.385941 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.386544 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.387062 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.388027 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.388227 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.388289 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.390588 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.390589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.390589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.390598 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.392032 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.392471 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.392472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.392473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.392541 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.392776 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.392777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.392777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.392943 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.393024 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.393294 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.393401 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:22:44.393755 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.384569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:45071 (local address 127.11.125.1:58819)
0925 02:22:44.384734 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:44.384778 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:44.385747 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:44.385999 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:44.386002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:44.386272 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:44.386470 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.386473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.387150 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.387151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.387950 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.387953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.388312 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.388313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.388411 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.388870 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:44.388879 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:44.390485 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:44.392132 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.392142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.392145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.392396 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.392611 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.392611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.392611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.392711 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.393011 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:44.393017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:44.393137 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 02:22:44.393274 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 02:22:44.393457 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260925 02:22:44.394371 15584 heartbeater.cc:347] Connected to a master server at 127.11.125.62:45071
I20260925 02:22:44.394480 15584 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:44.394692 15584 heartbeater.cc:511] Master 127.11.125.62:45071 requested a full tablet report, sending...
I20260925 02:22:44.395303 15392 ts_manager.cc:194] Registered new tserver with Master: 531cdc1726f5430fa39ff8417f6992bd (127.11.125.1:40987)
I20260925 02:22:44.395939 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:58819
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:44.416404 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:44.501145 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:44.501348 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:44.501371 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:44.502559 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:44.502609 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:44.502663 15589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:44.504351 15589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:44.504611 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:44.504843 15589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:44.509778 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.509795 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.509853 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:44.510099 15589 server_base.cc:1034] running on GCE node
I20260925 02:22:44.510291 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:44.510483 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:44.511632 15589 hybrid_clock.cc:648] HybridClock initialized: now 1790302964511619 us; error 26 us; skew 500 ppm
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:44.514173 15589 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:44.514659 15589 webserver.cc:460] Webserver started at http://127.11.125.2:39753/ using document root <none> and password file <none>
I20260925 02:22:44.514848 15589 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:44.514889 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:44.514977 15589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:44.515946 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "a211df548d1d432589c3f1aaa4fd41b8"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "9bf1c4d161840f3988c4bfca5c852362"
server_key_iv: "d1bde2fcfc1b3aecefed8a08959bdd81"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.516250 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "a211df548d1d432589c3f1aaa4fd41b8"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "9bf1c4d161840f3988c4bfca5c852362"
server_key_iv: "d1bde2fcfc1b3aecefed8a08959bdd81"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.517586 15589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:44.518265 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.518442 15589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:44.518550 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "a211df548d1d432589c3f1aaa4fd41b8"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "9bf1c4d161840f3988c4bfca5c852362"
server_key_iv: "d1bde2fcfc1b3aecefed8a08959bdd81"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.518630 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:44.527590 15589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:44.529387 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:44.529541 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:44.529759 15589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:44.530053 15589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:44.530113 15589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.530155 15589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:44.530197 15589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.536144 15589 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:39589
I20260925 02:22:44.536195 15718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:39589 every 8 connection(s)
I20260925 02:22:44.536495 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:44.541601 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15589
I20260925 02:22:44.541747 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:44.541939 11764 external_mini_cluster.cc:1428] Setting key b1dbeefb4bae2513a2ee95e076af0948
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:44.546376 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.537258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:58447 (local address 127.11.125.62:45071)
0925 02:22:44.537335 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.537338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.537514 (+ 176us) server_negotiation.cc:407] Connection header received
0925 02:22:44.538517 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.538518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.538530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.538656 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.539243 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.539788 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.540804 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.541061 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.541120 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.543491 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.543493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.543493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.543500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.544972 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.545392 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.545394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.545394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.545459 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.545717 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.545718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.545719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.545856 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.545913 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.546203 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.546287 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:22:44.546566 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.537204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:45071 (local address 127.11.125.2:58447)
0925 02:22:44.537382 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:44.537419 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:44.538404 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:44.538719 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:44.538723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:44.538971 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:44.539154 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.539158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.539895 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.539896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.540728 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.540730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.541160 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.541161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.541279 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.541873 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:44.541886 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:44.543394 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:44.545083 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.545090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.545092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.545314 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.545528 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.545529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.545529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.545641 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.545916 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:44.545923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:44.546060 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 02:22:44.546170 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:22:44.546325 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260925 02:22:44.547070 15719 heartbeater.cc:347] Connected to a master server at 127.11.125.62:45071
I20260925 02:22:44.547185 15719 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:44.547406 15719 heartbeater.cc:511] Master 127.11.125.62:45071 requested a full tablet report, sending...
I20260925 02:22:44.547837 15392 ts_manager.cc:194] Registered new tserver with Master: a211df548d1d432589c3f1aaa4fd41b8 (127.11.125.2:39589)
I20260925 02:22:44.548372 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:58447
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:44.567385 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:44.651907 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:44.652132 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:44.652158 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:44.653415 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:44.653463 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:44.653514 15724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:44.655095 15724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:44.655354 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:44.655606 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:44.660801 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:44.660817 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:44.661033 15724 server_base.cc:1034] running on GCE node
W20260925 02:22:44.661128 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:44.661327 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:44.661535 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:44.662678 15724 hybrid_clock.cc:648] HybridClock initialized: now 1790302964662649 us; error 49 us; skew 500 ppm
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:44.665194 15724 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:44.665632 15724 webserver.cc:460] Webserver started at http://127.11.125.3:36791/ using document root <none> and password file <none>
I20260925 02:22:44.665864 15724 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:44.665933 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:44.666086 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:44.667150 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "fd2a5ee832284ac1813a398d7930b759"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "81bd97dc52f3b706401a58c77aa9391d"
server_key_iv: "59cef09824e000944cc95bdfbbf10340"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.667512 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "fd2a5ee832284ac1813a398d7930b759"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "81bd97dc52f3b706401a58c77aa9391d"
server_key_iv: "59cef09824e000944cc95bdfbbf10340"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.668844 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:44.669613 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.669793 15724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:44.669893 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "fd2a5ee832284ac1813a398d7930b759"
format_stamp: "Formatted at 2026-09-25 02:22:44 on dist-test-slave-hj7s"
server_key: "81bd97dc52f3b706401a58c77aa9391d"
server_key_iv: "59cef09824e000944cc95bdfbbf10340"
server_key_version: "encryptionkey@0"
I20260925 02:22:44.669977 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:44.698779 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:44.700717 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:44.700937 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:44.701208 15724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:44.701537 15724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:44.701593 15724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.701640 15724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:44.701664 15724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:44.708266 15724 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:43197
I20260925 02:22:44.708316 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:43197 every 8 connection(s)
I20260925 02:22:44.708637 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:44.712651 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15724
I20260925 02:22:44.712780 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:44.712998 11764 external_mini_cluster.cc:1428] Setting key ab97bdf678d99d2c6a3072ed50831337
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:44.717845 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.709381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:47527 (local address 127.11.125.62:45071)
0925 02:22:44.709451 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.709454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.709684 (+ 230us) server_negotiation.cc:407] Connection header received
0925 02:22:44.710602 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.710603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.710613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.710669 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.711266 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.711768 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.712726 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.713064 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.713136 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.715431 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.715432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.715433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.715439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.716684 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.717125 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.717126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.717126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.717172 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.717380 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.717380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.717380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.717471 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.717517 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.717725 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.717780 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:22:44.718060 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.709368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:45071 (local address 127.11.125.3:47527)
0925 02:22:44.709549 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:44.709594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:44.710512 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:44.710732 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:44.710735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:44.710998 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:44.711192 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.711196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.711850 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.711851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.712637 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:44.712639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.713164 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.713166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:44.713323 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.713887 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:44.713898 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:44.715343 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:44.716766 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.716773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.716774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.717064 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.717229 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:44.717230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:44.717230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:44.717333 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:44.717522 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:44.717526 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:44.717635 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 02:22:44.717714 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:22:44.717841 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260925 02:22:44.718545 15853 heartbeater.cc:347] Connected to a master server at 127.11.125.62:45071
I20260925 02:22:44.718658 15853 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:44.718849 15853 heartbeater.cc:511] Master 127.11.125.62:45071 requested a full tablet report, sending...
I20260925 02:22:44.719229 15392 ts_manager.cc:194] Registered new tserver with Master: fd2a5ee832284ac1813a398d7930b759 (127.11.125.3:43197)
I20260925 02:22:44.719617 15392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:47527
I20260925 02:22:44.724884 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:44.748153 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.740546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.11.125.1:40987)
0925 02:22:44.740739 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.740744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.740757 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:44.740787 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.740789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.740803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.740871 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.741495 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.742204 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.743015 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.743334 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.743387 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.745553 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.745562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.745562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.745569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.747183 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.747478 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.747480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.747481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.747534 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.747678 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.747678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.747679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.747795 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.747889 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.747961 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.748019 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:44.755817 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:44.749491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.11.125.62:45071)
0925 02:22:44.749538 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:44.749540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:44.749552 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:44.749581 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:44.749581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:44.749590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:44.749634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:44.750093 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.750601 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.751224 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:44.751416 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:44.751465 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:44.753548 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:44.753550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:44.753550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:44.753556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:44.754957 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.755234 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.755234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.755235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.755296 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:44.755436 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:44.755436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:44.755436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:44.755566 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:44.755631 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:44.755695 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:22:44.755753 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:22:44.756059 15392 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51960
I20260925 02:22:44.756407 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15454
I20260925 02:22:44.761881 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15589
I20260925 02:22:44.766726 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15724
I20260925 02:22:44.771724 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15360
2026-09-25T02:22:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1878 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15865](info): set up 2 sockets
Sep 25 02:22:44 dist-test-slave-hj7s krb5kdc[15865](info): commencing operation
krb5kdc: starting...
W20260925 02:22:45.815068 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:45 dist-test-slave-hj7s krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:45.943677 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33417
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:46.027252 15879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:46.027467 15879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:46.027495 15879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:46.028765 15879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:46.028807 15879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:46.028821 15879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:46.028834 15879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:46.028887 15879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:46.030507 15879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:33417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:46.030782 15879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:46.031051 15879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:46.036576 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.036614 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.036614 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.036640 15879 server_base.cc:1034] running on GCE node
I20260925 02:22:46.037055 15879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:46.037345 15879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:46.038489 15879 hybrid_clock.cc:648] HybridClock initialized: now 1790302966038466 us; error 39 us; skew 500 ppm
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:46.041574 15879 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:46.041988 15879 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:46.042063 15879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:46.042217 15879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:46.043300 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "70ccdf5bc18449d38c92975e696ab91e"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "c1cb62743368c5fa0295d4c768fbe6fd"
server_key_iv: "9ee0be55cdc8903d41d9d353293c67f7"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.043663 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "70ccdf5bc18449d38c92975e696ab91e"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "c1cb62743368c5fa0295d4c768fbe6fd"
server_key_iv: "9ee0be55cdc8903d41d9d353293c67f7"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.045039 15879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:46.045792 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.046068 15879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:46.046203 15879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "70ccdf5bc18449d38c92975e696ab91e"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "c1cb62743368c5fa0295d4c768fbe6fd"
server_key_iv: "9ee0be55cdc8903d41d9d353293c67f7"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.046303 15879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:46.080271 15879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:46.083406 15879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:46.083577 15879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:46.089241 15879 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:33417
I20260925 02:22:46.089259 15946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:33417 every 8 connection(s)
I20260925 02:22:46.089632 15879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:46.090745 15947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:46.093359 15947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: Bootstrap starting.
I20260925 02:22:46.093921 15947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:46.094174 15947 log.cc:824] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:46.094791 15947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: No bootstrap required, opened a new log
I20260925 02:22:46.096323 15947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } }
I20260925 02:22:46.096463 15947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:46.096487 15947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ccdf5bc18449d38c92975e696ab91e, State: Initialized, Role: FOLLOWER
I20260925 02:22:46.096632 15947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } }
I20260925 02:22:46.096711 15947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:46.096735 15947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:46.096762 15947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:46.097613 15947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } }
I20260925 02:22:46.097710 15947 leader_election.cc:302] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ccdf5bc18449d38c92975e696ab91e; no voters:
I20260925 02:22:46.097839 15947 leader_election.cc:288] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:46.097909 15950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:46.098095 15950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [term 1 LEADER]: Becoming Leader. State: Replica: 70ccdf5bc18449d38c92975e696ab91e, State: Running, Role: LEADER
I20260925 02:22:46.098142 15947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:46.098191 15950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } }
I20260925 02:22:46.098572 15952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ccdf5bc18449d38c92975e696ab91e. Latest consensus state: current_term: 1 leader_uuid: "70ccdf5bc18449d38c92975e696ab91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } } }
I20260925 02:22:46.098670 15952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:46.098565 15951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ccdf5bc18449d38c92975e696ab91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ccdf5bc18449d38c92975e696ab91e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33417 } } }
I20260925 02:22:46.098824 15951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:46.098891 15956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:46.099268 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15879
I20260925 02:22:46.099398 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:46.099529 15956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:46.099575 11764 external_mini_cluster.cc:1428] Setting key ebe1485e1942efd028bffeed42d1ccd7
I20260925 02:22:46.101354 15956 catalog_manager.cc:1347] Generated new cluster ID: 3e5ddeb00678447c92edc6301669b9b9
I20260925 02:22:46.101413 15956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:46.107717 15956 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:46.108237 15956 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:46.110491 15956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: Generated new TSK 0
I20260925 02:22:46.110658 15956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:46.112102 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.100019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.11.125.62:33417)
0925 02:22:46.100749 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.100764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.100819 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:22:46.100973 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.100981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.101041 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.101185 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:46.101987 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.103363 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.104476 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.105171 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.105277 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.108371 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:46.108375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:46.108375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:46.108388 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:46.110503 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.110953 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.110955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.110956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.111038 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.111251 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.111256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.111256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.111453 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:46.111567 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.111706 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.111822 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:46.140175 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:46.260465 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:46.260720 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:46.260762 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:46.262116 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:46.262192 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:46.262243 15973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:46.263990 15973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:46.264287 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:46.264551 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:46.270267 15982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.270284 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.270402 15973 server_base.cc:1034] running on GCE node
W20260925 02:22:46.270284 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.270802 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:46.271003 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:46.272141 15973 hybrid_clock.cc:648] HybridClock initialized: now 1790302966272126 us; error 31 us; skew 500 ppm
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:46.274699 15973 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:46.275056 15973 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:46.275105 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:46.275195 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:46.276225 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "cb25c174f6e64ee1810ddb68c7095340"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "6bbc89938e02d9679223e13247bfb304"
server_key_iv: "42ab592c35e8868c377457da8d2b69d0"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.276597 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "cb25c174f6e64ee1810ddb68c7095340"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "6bbc89938e02d9679223e13247bfb304"
server_key_iv: "42ab592c35e8868c377457da8d2b69d0"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.277918 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:46.278614 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.278811 15973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:46.278905 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "cb25c174f6e64ee1810ddb68c7095340"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "6bbc89938e02d9679223e13247bfb304"
server_key_iv: "42ab592c35e8868c377457da8d2b69d0"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.278975 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:46.309252 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:46.311825 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:46.312007 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:46.312290 15973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:46.312651 15973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:46.312708 15973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.312743 15973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:46.312772 15973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.318819 15973 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37307
I20260925 02:22:46.318902 16102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37307 every 8 connection(s)
I20260925 02:22:46.319181 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:46.327279 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 15973
I20260925 02:22:46.327373 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:46.327608 11764 external_mini_cluster.cc:1428] Setting key 4196a3b9a428f34db809cb186d95992e
I20260925 02:22:46.328744 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.319979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:43615 (local address 127.11.125.62:33417)
0925 02:22:46.320030 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.320033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.320233 (+ 200us) server_negotiation.cc:407] Connection header received
0925 02:22:46.320791 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.320792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.320806 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.320867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:46.321860 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.322373 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.323399 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.323610 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.323719 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.325837 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:46.325839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:46.325839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:46.325848 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:46.327032 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.327504 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.327505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.327506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.327585 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.327935 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.327936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.327937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.328085 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:46.328178 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.328551 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.328653 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:46.328995 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.319941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33417 (local address 127.11.125.1:43615)
0925 02:22:46.320088 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.320138 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.320669 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.320957 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.320961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.321237 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:46.321746 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.321751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.322476 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.322477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.323298 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.323300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.323729 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.323731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.323870 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.324340 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:46.324349 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:46.325734 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:46.327124 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.327133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.327136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.327414 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.327684 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.327685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.327686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.327839 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.328170 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:46.328179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:46.328386 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.328526 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.328692 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 02:22:46.329464 16103 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33417
I20260925 02:22:46.329566 16103 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:46.329723 16103 heartbeater.cc:511] Master 127.11.125.62:33417 requested a full tablet report, sending...
I20260925 02:22:46.330307 15911 ts_manager.cc:194] Registered new tserver with Master: cb25c174f6e64ee1810ddb68c7095340 (127.11.125.1:37307)
I20260925 02:22:46.330906 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:43615
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:46.355656 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:46.439644 16108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:46.439852 16108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:46.439877 16108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:46.441170 16108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:46.441214 16108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:46.441242 16108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:46.442792 16108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:46.443039 16108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:46.443287 16108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:46.448482 16113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.448454 16114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.448614 16108 server_base.cc:1034] running on GCE node
W20260925 02:22:46.448454 16116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.449033 16108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:46.449308 16108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:46.450474 16108 hybrid_clock.cc:648] HybridClock initialized: now 1790302966450461 us; error 45 us; skew 500 ppm
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:46.453101 16108 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:46.453470 16108 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:46.453542 16108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:46.453696 16108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:46.454813 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "f76b3202aada781a959868c660884488"
server_key_iv: "32a12995af62ec1e3696505174cd9ed4"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.455183 16108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "f76b3202aada781a959868c660884488"
server_key_iv: "32a12995af62ec1e3696505174cd9ed4"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.456568 16108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:46.457355 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.457522 16108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.457631 16108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "f76b3202aada781a959868c660884488"
server_key_iv: "32a12995af62ec1e3696505174cd9ed4"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.457708 16108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:46.475121 16108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:46.477169 16108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:46.477425 16108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:46.477770 16108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:46.478140 16108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:46.478204 16108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.478250 16108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:46.478276 16108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.484324 16108 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36627
I20260925 02:22:46.484401 16236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36627 every 8 connection(s)
I20260925 02:22:46.484691 16108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:46.490832 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16108
I20260925 02:22:46.490912 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:46.491122 11764 external_mini_cluster.cc:1428] Setting key dd41182880f05230bfb242ec4aa26ea2
I20260925 02:22:46.494122 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.485475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:54521 (local address 127.11.125.62:33417)
0925 02:22:46.485569 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.485571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.485734 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:22:46.486202 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.486204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.486214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.486269 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:46.487176 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.487735 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.488693 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.488881 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.489009 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.491172 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:46.491174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:46.491175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:46.491182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:46.492624 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.493111 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.493113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.493113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.493179 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.493446 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.493448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.493448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.493566 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:46.493644 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.493947 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.494033 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:22:46.494329 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.485423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33417 (local address 127.11.125.2:54521)
0925 02:22:46.485611 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.485648 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.486112 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.486323 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.486327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.486582 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:46.487084 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.487088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.487819 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.487820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.488606 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.488608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.489017 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.489018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.489120 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.489593 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:46.489600 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:46.491076 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:46.492713 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.492722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.492724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.493035 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.493263 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.493263 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.493264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.493378 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.493646 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:46.493654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:46.493800 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.493913 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.494073 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260925 02:22:46.494889 16237 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33417
I20260925 02:22:46.495047 16237 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:46.495235 16237 heartbeater.cc:511] Master 127.11.125.62:33417 requested a full tablet report, sending...
I20260925 02:22:46.495721 15911 ts_manager.cc:194] Registered new tserver with Master: 8ca6b68f5ef246b7ae0926c44d30ebbb (127.11.125.2:36627)
I20260925 02:22:46.496114 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:54521
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:46.519706 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:46.602478 16242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:46.602689 16242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:46.602715 16242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:46.603832 16242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:46.603866 16242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:46.603892 16242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:46.605501 16242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:46.605696 16242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:46.605873 16242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:46.610968 16248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.610941 16250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:46.611166 16247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:46.611153 16242 server_base.cc:1034] running on GCE node
I20260925 02:22:46.611478 16242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:46.611748 16242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:46.612952 16242 hybrid_clock.cc:648] HybridClock initialized: now 1790302966612923 us; error 58 us; skew 500 ppm
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:46.615605 16242 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:46.615916 16242 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:46.615960 16242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:46.616050 16242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:46.617149 16242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "50681ca1991f46989bed29e55ec60076"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "373882b5a4e7a78796ec3307e3cf3830"
server_key_iv: "b74c94de336722cf13d1554a137d61c3"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.617461 16242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "50681ca1991f46989bed29e55ec60076"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "373882b5a4e7a78796ec3307e3cf3830"
server_key_iv: "b74c94de336722cf13d1554a137d61c3"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.618686 16242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:46.619377 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.619554 16242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:46.619644 16242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "50681ca1991f46989bed29e55ec60076"
format_stamp: "Formatted at 2026-09-25 02:22:46 on dist-test-slave-hj7s"
server_key: "373882b5a4e7a78796ec3307e3cf3830"
server_key_iv: "b74c94de336722cf13d1554a137d61c3"
server_key_version: "encryptionkey@0"
I20260925 02:22:46.619712 16242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:46.638871 16242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:46.641008 16242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:46.641176 16242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:46.641440 16242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:46.641803 16242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:46.641867 16242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.641914 16242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:46.641937 16242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:46.647828 16242 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:37569
I20260925 02:22:46.647953 16370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:37569 every 8 connection(s)
I20260925 02:22:46.648208 16242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:22:46 dist-test-slave-hj7s krb5kdc[15865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302966, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:46.654721 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16242
I20260925 02:22:46.654825 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:46.655076 11764 external_mini_cluster.cc:1428] Setting key 1d12a89f8ecd8dadbcc6192dc9e5121a
I20260925 02:22:46.657791 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.649023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:43269 (local address 127.11.125.62:33417)
0925 02:22:46.649071 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.649073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.649380 (+ 307us) server_negotiation.cc:407] Connection header received
0925 02:22:46.649872 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.649873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.649883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.649944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:46.650952 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.651464 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.652469 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.652654 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.652707 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.655026 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:46.655028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:46.655028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:46.655037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:46.656551 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.656977 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.656978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.656978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.657033 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.657253 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.657254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.657254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.657360 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:46.657410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.657648 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.657714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:22:46.657963 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.649079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33417 (local address 127.11.125.3:43269)
0925 02:22:46.649245 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.649290 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.649776 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.650012 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.650015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.650345 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:46.650865 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.650870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.651555 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.651555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.652388 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.652390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.652744 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.652745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.652861 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.653421 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:46.653430 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:46.654875 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:46.656626 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.656633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.656635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.656861 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.657092 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:46.657093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:46.657093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:46.657197 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:46.657414 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:46.657419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:46.657536 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.657620 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.657744 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260925 02:22:46.658495 16371 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33417
I20260925 02:22:46.658610 16371 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:46.658771 16371 heartbeater.cc:511] Master 127.11.125.62:33417 requested a full tablet report, sending...
I20260925 02:22:46.659186 15911 ts_manager.cc:194] Registered new tserver with Master: 50681ca1991f46989bed29e55ec60076 (127.11.125.3:37569)
I20260925 02:22:46.659583 15911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:43269
I20260925 02:22:46.666836 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:22:46.674547 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.668259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.11.125.62:33417)
0925 02:22:46.668323 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.668325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.668397 (+ 72us) server_negotiation.cc:407] Connection header received
0925 02:22:46.668440 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.668440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.668449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.668505 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:46.669181 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.669665 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.670360 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.670542 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.670596 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.671728 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:46.671729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:46.671730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:46.671736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:46.673203 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.673498 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.673499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.673499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.673573 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:46.673794 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:46.673794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:46.673795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:46.674032 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:46.674149 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.674318 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.674420 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:46.676101 15911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:46.677784 15911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:46.684824 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.11.125.1:37307)
0925 02:22:46.680863 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.680867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.680878 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:22:46.680936 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.680939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.680956 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.681032 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:46.681187 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682276 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.683917 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.684423 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.684514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.684579 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.684688 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.684728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":96,"threads_started":1}
I20260925 02:22:46.684986 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36627 (local address 127.0.0.1:46542)
0925 02:22:46.680570 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.680586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.680662 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.680891 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.680894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.680896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:46.680998 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.681000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682612 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.682613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.683578 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.683581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.684633 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.684635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.684713 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.684717 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.684766 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.684805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":61,"threads_started":1}
I20260925 02:22:46.685075 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.11.125.2:36627)
0925 02:22:46.680494 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.680500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.680651 (+ 151us) server_negotiation.cc:407] Connection header received
0925 02:22:46.680733 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.680736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.680752 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.680868 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:46.681088 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682285 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.683676 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.684462 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.684536 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.684592 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.684910 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.684978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260925 02:22:46.685256 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:37569 (local address 127.0.0.1:51104)
0925 02:22:46.681198 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.681209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.681265 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.681442 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.681443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.681445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:46.681493 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.681495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682469 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.682470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.683812 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.683815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.685008 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.685010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.685065 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.685069 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.685121 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.685161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":39,"thread_start_us":68,"threads_started":1}
I20260925 02:22:46.684815 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37307 (local address 127.0.0.1:36672)
0925 02:22:46.680512 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.680557 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.680631 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.681044 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.681045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.681047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:46.681100 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.681101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682389 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.682390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.683817 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.683819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.684513 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.684514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.684593 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.684607 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.684676 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.684728 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":92,"thread_start_us":102,"threads_started":1}
I20260925 02:22:46.685508 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.680556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.11.125.3:37569)
0925 02:22:46.680743 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.680747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.681266 (+ 519us) server_negotiation.cc:407] Connection header received
0925 02:22:46.681334 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.681337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.681352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.681446 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:46.681558 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.682365 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.683907 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.684613 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.684687 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.684739 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.685374 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.685421 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260925 02:22:46.685937 16170 tablet_service.cc:1467] Processing CreateTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 (DEFAULT_TABLE table=test-table [id=9099956ef5f24245ad8d9cb741149be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:46.686117 16304 tablet_service.cc:1467] Processing CreateTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 (DEFAULT_TABLE table=test-table [id=9099956ef5f24245ad8d9cb741149be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:46.686374 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b6cc11b4c04c6184fb694d39848da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:46.686509 16304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b6cc11b4c04c6184fb694d39848da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:46.689111 16389 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Bootstrap starting.
I20260925 02:22:46.689116 16388 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Bootstrap starting.
I20260925 02:22:46.689630 16036 tablet_service.cc:1467] Processing CreateTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 (DEFAULT_TABLE table=test-table [id=9099956ef5f24245ad8d9cb741149be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:46.689837 16388 tablet_bootstrap.cc:654] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:46.689932 16389 tablet_bootstrap.cc:654] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:46.690003 16036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b6cc11b4c04c6184fb694d39848da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:46.690444 16388 log.cc:824] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:46.690464 16389 log.cc:824] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:46.691243 16388 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: No bootstrap required, opened a new log
I20260925 02:22:46.691324 16389 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: No bootstrap required, opened a new log
I20260925 02:22:46.691336 16388 ts_tablet_manager.cc:1397] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:46.691397 16389 ts_tablet_manager.cc:1397] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:46.693181 16388 raft_consensus.cc:348] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.693342 16388 raft_consensus.cc:374] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:46.693387 16388 raft_consensus.cc:729] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca6b68f5ef246b7ae0926c44d30ebbb, State: Initialized, Role: FOLLOWER
I20260925 02:22:46.693531 16388 consensus_queue.cc:260] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.693553 16389 raft_consensus.cc:348] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.693722 16389 raft_consensus.cc:374] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:46.693790 16389 raft_consensus.cc:729] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50681ca1991f46989bed29e55ec60076, State: Initialized, Role: FOLLOWER
I20260925 02:22:46.693851 16388 ts_tablet_manager.cc:1428] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:22:46.693894 16392 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Bootstrap starting.
I20260925 02:22:46.693915 16389 consensus_queue.cc:260] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.693969 16237 heartbeater.cc:503] Master 127.11.125.62:33417 was elected leader, sending a full tablet report...
I20260925 02:22:46.694204 16389 ts_tablet_manager.cc:1428] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:46.694213 16371 heartbeater.cc:503] Master 127.11.125.62:33417 was elected leader, sending a full tablet report...
I20260925 02:22:46.694553 16392 tablet_bootstrap.cc:654] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:46.695183 16392 log.cc:824] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:46.696067 16392 tablet_bootstrap.cc:492] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: No bootstrap required, opened a new log
I20260925 02:22:46.696131 16392 ts_tablet_manager.cc:1397] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:46.697625 16392 raft_consensus.cc:348] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.697767 16392 raft_consensus.cc:374] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:46.697790 16392 raft_consensus.cc:729] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb25c174f6e64ee1810ddb68c7095340, State: Initialized, Role: FOLLOWER
I20260925 02:22:46.697897 16392 consensus_queue.cc:260] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.698129 16392 ts_tablet_manager.cc:1428] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:46.698206 16103 heartbeater.cc:503] Master 127.11.125.62:33417 was elected leader, sending a full tablet report...
W20260925 02:22:46.735273 16238 tablet.cc:2369] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:22:46.819995 16104 tablet.cc:2369] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:22:46.898793 16372 tablet.cc:2369] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:46.993577 16394 raft_consensus.cc:482] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:46.993726 16394 raft_consensus.cc:504] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.994099 16394 leader_election.cc:288] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb25c174f6e64ee1810ddb68c7095340 (127.11.125.1:37307), 8ca6b68f5ef246b7ae0926c44d30ebbb (127.11.125.2:36627)
I20260925 02:22:46.997010 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.994284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37307 (local address 127.11.125.3:51691)
0925 02:22:46.994349 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.994361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.994413 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.994594 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.994595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.994597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:46.994673 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.994675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.995351 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.995352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.996122 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.996124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.996697 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.996789 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.996792 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.996858 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.996903 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:22:46.997010 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.994367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:51691 (local address 127.11.125.1:37307)
0925 02:22:46.994445 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.994447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.994458 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:22:46.994493 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.994494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.994504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.994575 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:46.994753 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.995244 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996235 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.996610 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996697 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.996707 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.996878 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.996943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:22:46.997287 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.994452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:52935 (local address 127.11.125.2:36627)
0925 02:22:46.994510 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:46.994512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:46.994752 (+ 240us) server_negotiation.cc:407] Connection header received
0925 02:22:46.994803 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:46.994804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:46.994814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:46.994888 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:46.995045 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.995520 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996475 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.996849 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996955 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.996964 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:46.997154 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:22:46.997210 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:22:46.997406 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b6cc11b4c04c6184fb694d39848da0" candidate_uuid: "50681ca1991f46989bed29e55ec60076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb25c174f6e64ee1810ddb68c7095340" is_pre_election: true
I20260925 02:22:46.997507 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:46.994356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36627 (local address 127.11.125.3:52935)
0925 02:22:46.994667 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:46.994676 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:46.994754 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:46.994890 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:46.994891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:46.994892 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:46.994973 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.994975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.995608 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.995609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.996378 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:46.996380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:46.996963 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:46.996964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:46.997037 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:46.997041 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:46.997371 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 02:22:46.997419 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":106,"threads_started":1}
I20260925 02:22:46.997605 16056 raft_consensus.cc:2454] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50681ca1991f46989bed29e55ec60076 in term 0.
I20260925 02:22:46.997864 16258 leader_election.cc:302] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50681ca1991f46989bed29e55ec60076, cb25c174f6e64ee1810ddb68c7095340; no voters:
I20260925 02:22:46.997864 16191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b6cc11b4c04c6184fb694d39848da0" candidate_uuid: "50681ca1991f46989bed29e55ec60076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" is_pre_election: true
I20260925 02:22:46.997994 16394 raft_consensus.cc:2781] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:46.998018 16191 raft_consensus.cc:2454] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50681ca1991f46989bed29e55ec60076 in term 0.
I20260925 02:22:46.998070 16394 raft_consensus.cc:482] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:46.998102 16394 raft_consensus.cc:3037] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:46.998927 16394 raft_consensus.cc:504] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:46.999081 16394 leader_election.cc:288] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [CANDIDATE]: Term 1 election: Requested vote from peers cb25c174f6e64ee1810ddb68c7095340 (127.11.125.1:37307), 8ca6b68f5ef246b7ae0926c44d30ebbb (127.11.125.2:36627)
I20260925 02:22:46.999228 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b6cc11b4c04c6184fb694d39848da0" candidate_uuid: "50681ca1991f46989bed29e55ec60076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb25c174f6e64ee1810ddb68c7095340"
I20260925 02:22:46.999284 16191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b6cc11b4c04c6184fb694d39848da0" candidate_uuid: "50681ca1991f46989bed29e55ec60076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb"
I20260925 02:22:46.999318 16056 raft_consensus.cc:3037] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:46.999390 16191 raft_consensus.cc:3037] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:47.000109 16056 raft_consensus.cc:2454] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50681ca1991f46989bed29e55ec60076 in term 1.
I20260925 02:22:47.000180 16191 raft_consensus.cc:2454] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50681ca1991f46989bed29e55ec60076 in term 1.
I20260925 02:22:47.000295 16258 leader_election.cc:302] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50681ca1991f46989bed29e55ec60076, cb25c174f6e64ee1810ddb68c7095340; no voters:
I20260925 02:22:47.000394 16394 raft_consensus.cc:2781] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:47.000556 16394 raft_consensus.cc:686] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 1 LEADER]: Becoming Leader. State: Replica: 50681ca1991f46989bed29e55ec60076, State: Running, Role: LEADER
I20260925 02:22:47.000656 16394 consensus_queue.cc:237] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } }
I20260925 02:22:47.001523 15911 catalog_manager.cc:5495] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50681ca1991f46989bed29e55ec60076 (127.11.125.3). New cstate: current_term: 1 leader_uuid: "50681ca1991f46989bed29e55ec60076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50681ca1991f46989bed29e55ec60076" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37569 } health_report { overall_health: HEALTHY } } }
I20260925 02:22:47.041591 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.038938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.11.125.3:37569)
0925 02:22:47.039036 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.039038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.039047 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:47.039086 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.039087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.039105 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.039172 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:47.039332 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:47.039804 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:47.040618 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:47.040801 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:47.040848 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:47.041232 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:47.041350 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:47.041420 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.041493 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.041512 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:22:47.043603 16191 raft_consensus.cc:1261] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 1 FOLLOWER]: Refusing update from remote peer 50681ca1991f46989bed29e55ec60076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:47.043606 16056 raft_consensus.cc:1261] T 31b6cc11b4c04c6184fb694d39848da0 P cb25c174f6e64ee1810ddb68c7095340 [term 1 FOLLOWER]: Refusing update from remote peer 50681ca1991f46989bed29e55ec60076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:47.043973 16394 consensus_queue.cc:1035] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca6b68f5ef246b7ae0926c44d30ebbb" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:47.044175 16398 consensus_queue.cc:1035] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb25c174f6e64ee1810ddb68c7095340" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:47.046622 16401 mvcc.cc:205] Tried to move back new op lower bound from 7333080953007284224 to 7333080952835215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333080953007284224}]
I20260925 02:22:47.047004 16402 mvcc.cc:205] Tried to move back new op lower bound from 7333080953007284224 to 7333080952835215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333080953007284224}]
I20260925 02:22:47.047360 16404 mvcc.cc:205] Tried to move back new op lower bound from 7333080953007284224 to 7333080952835215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333080953007284224}]
I20260925 02:22:47.049062 15910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:47.049185 15910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:47.049997 15910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70ccdf5bc18449d38c92975e696ab91e: Sending DeleteTablet for 3 replicas of tablet 31b6cc11b4c04c6184fb694d39848da0
I20260925 02:22:47.050400 16170 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46542
I20260925 02:22:47.050464 16036 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:36672
I20260925 02:22:47.050649 16304 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b6cc11b4c04c6184fb694d39848da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:51104
I20260925 02:22:47.050666 16410 tablet_replica.cc:330] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: stopping tablet replica
I20260925 02:22:47.050766 16410 raft_consensus.cc:2229] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:47.050886 16410 raft_consensus.cc:2258] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:47.050902 16412 tablet_replica.cc:330] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: stopping tablet replica
I20260925 02:22:47.051082 16412 raft_consensus.cc:2229] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:47.051170 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15973
I20260925 02:22:47.051321 16412 raft_consensus.cc:2258] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:47.051438 16410 ts_tablet_manager.cc:1905] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:47.051872 16412 ts_tablet_manager.cc:1905] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:47.053143 16410 ts_tablet_manager.cc:1918] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:47.053224 16410 log.cc:1196] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/31b6cc11b4c04c6184fb694d39848da0
I20260925 02:22:47.053481 16410 ts_tablet_manager.cc:1939] T 31b6cc11b4c04c6184fb694d39848da0 P 8ca6b68f5ef246b7ae0926c44d30ebbb: Deleting consensus metadata
I20260925 02:22:47.054042 15894 catalog_manager.cc:4841] TS 8ca6b68f5ef246b7ae0926c44d30ebbb (127.11.125.2:36627): tablet 31b6cc11b4c04c6184fb694d39848da0 (table test-table [id=9099956ef5f24245ad8d9cb741149be2]) successfully deleted
I20260925 02:22:47.054419 16412 ts_tablet_manager.cc:1918] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:47.054548 16412 log.cc:1196] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/31b6cc11b4c04c6184fb694d39848da0
I20260925 02:22:47.054890 16412 ts_tablet_manager.cc:1939] T 31b6cc11b4c04c6184fb694d39848da0 P 50681ca1991f46989bed29e55ec60076: Deleting consensus metadata
I20260925 02:22:47.055418 15894 catalog_manager.cc:4841] TS 50681ca1991f46989bed29e55ec60076 (127.11.125.3:37569): tablet 31b6cc11b4c04c6184fb694d39848da0 (table test-table [id=9099956ef5f24245ad8d9cb741149be2]) successfully deleted
W20260925 02:22:47.058934 15895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:37307 (error 108)
I20260925 02:22:47.059188 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16108
I20260925 02:22:47.059433 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.059308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37307 (local address 127.0.0.1:36686)
0925 02:22:47.059358 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.059392 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:22:47.059543 15895 catalog_manager.cc:4584] TS cb25c174f6e64ee1810ddb68c7095340 (127.11.125.1:37307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31b6cc11b4c04c6184fb694d39848da0: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37307: connect: Connection refused (error 111)
I20260925 02:22:47.065092 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16242
I20260925 02:22:47.069916 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 15879
2026-09-25T02:22:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2297 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:47Z Disabled control of system clock
I20260925 02:22:47.096582 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36745
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:22:47.178723 16417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:47.178931 16417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:47.178952 16417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:47.180114 16417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:47.180158 16417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:47.180171 16417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:47.180183 16417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:47.180220 16417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:47.181641 16417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:36745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:47.181912 16417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:47.182215 16417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:47.187071 16425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.187206 16423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.187085 16422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.187346 16417 server_base.cc:1034] running on GCE node
I20260925 02:22:47.187610 16417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:47.187863 16417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:47.188970 16417 hybrid_clock.cc:648] HybridClock initialized: now 1790302967188956 us; error 29 us; skew 500 ppm
I20260925 02:22:47.190183 16417 webserver.cc:460] Webserver started at http://127.11.125.62:42249/ using document root <none> and password file <none>
I20260925 02:22:47.190449 16417 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:47.190515 16417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:47.190678 16417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:47.191762 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "b11ba4d2fa4742cabab74ea031696ebb"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "bece63ce6b16e3a6633783f4eec2f0df"
server_key_iv: "282ae838806fca968310207a3c0f8f12"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.192131 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "b11ba4d2fa4742cabab74ea031696ebb"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "bece63ce6b16e3a6633783f4eec2f0df"
server_key_iv: "282ae838806fca968310207a3c0f8f12"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.193516 16417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:47.194216 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.194406 16417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:47.194473 16417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "b11ba4d2fa4742cabab74ea031696ebb"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "bece63ce6b16e3a6633783f4eec2f0df"
server_key_iv: "282ae838806fca968310207a3c0f8f12"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.194561 16417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:47.197883 16417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:47.198150 16417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:47.198305 16417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:47.202203 16417 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:36745
I20260925 02:22:47.202231 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:36745 every 8 connection(s)
I20260925 02:22:47.202582 16417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:47.203222 16484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:47.206048 16484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: Bootstrap starting.
I20260925 02:22:47.206580 16484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:47.206838 16484 log.cc:824] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:47.207453 16484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: No bootstrap required, opened a new log
I20260925 02:22:47.209071 16484 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } }
I20260925 02:22:47.209208 16484 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:47.209231 16484 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11ba4d2fa4742cabab74ea031696ebb, State: Initialized, Role: FOLLOWER
I20260925 02:22:47.209360 16484 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } }
I20260925 02:22:47.209435 16484 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:47.209458 16484 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:47.209486 16484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:47.209967 16484 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } }
I20260925 02:22:47.210053 16484 leader_election.cc:302] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11ba4d2fa4742cabab74ea031696ebb; no voters:
I20260925 02:22:47.210175 16484 leader_election.cc:288] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:47.210255 16488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:47.210428 16488 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [term 1 LEADER]: Becoming Leader. State: Replica: b11ba4d2fa4742cabab74ea031696ebb, State: Running, Role: LEADER
I20260925 02:22:47.210459 16484 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:47.210525 16488 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } }
I20260925 02:22:47.210830 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16417
I20260925 02:22:47.210927 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:47.211140 11764 external_mini_cluster.cc:1428] Setting key 94e449e4413cc98c491da9dec4e8daf5
I20260925 02:22:47.211660 16490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11ba4d2fa4742cabab74ea031696ebb. Latest consensus state: current_term: 1 leader_uuid: "b11ba4d2fa4742cabab74ea031696ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } } }
I20260925 02:22:47.211755 16490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:47.211828 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11ba4d2fa4742cabab74ea031696ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ba4d2fa4742cabab74ea031696ebb" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36745 } } }
I20260925 02:22:47.211910 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:47.211963 16502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:47.212486 16502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:47.214088 16502 catalog_manager.cc:1347] Generated new cluster ID: 0731a1b6ff584e4082b0ceed8f2f649d
I20260925 02:22:47.214160 16502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:47.214331 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.211713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.11.125.62:36745)
0925 02:22:47.212114 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.212122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.212160 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:22:47.212452 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.212457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.212492 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.212627 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.213763 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.213766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.213767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.213771 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.213819 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.213831 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.213903 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.213976 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.214038 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
I20260925 02:22:47.223865 16502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:47.224018 16502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:47.233448 16502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: Generated new TSK 0
I20260925 02:22:47.233660 16502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:47.238900 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--builtin_ntp_servers=127.11.125.20:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:22:47.321543 16509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:47.321754 16509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:47.321777 16509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:47.322965 16509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:47.323005 16509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:47.323031 16509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:47.324421 16509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:47.324682 16509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:47.325050 16509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:47.330300 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.330431 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.330598 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.330578 16509 server_base.cc:1034] running on GCE node
I20260925 02:22:47.330883 16509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:47.331147 16509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:47.332310 16509 hybrid_clock.cc:648] HybridClock initialized: now 1790302967332288 us; error 37 us; skew 500 ppm
I20260925 02:22:47.333657 16509 webserver.cc:460] Webserver started at http://127.11.125.1:36235/ using document root <none> and password file <none>
I20260925 02:22:47.333935 16509 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:47.333982 16509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:47.334128 16509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:47.335212 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "47cddb96426c40919d358f159899c0e8"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "75e5093cf85604f10ab4a4791c86a031"
server_key_iv: "4f228e234a91a120812e482b1bebcbc2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.335621 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "47cddb96426c40919d358f159899c0e8"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "75e5093cf85604f10ab4a4791c86a031"
server_key_iv: "4f228e234a91a120812e482b1bebcbc2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.337010 16509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 02:22:47.337833 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.338027 16509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:47.338130 16509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "47cddb96426c40919d358f159899c0e8"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "75e5093cf85604f10ab4a4791c86a031"
server_key_iv: "4f228e234a91a120812e482b1bebcbc2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.338202 16509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:47.342794 16509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:47.343091 16509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:47.343219 16509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:47.343439 16509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:47.343798 16509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:47.343843 16509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.343873 16509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:47.343895 16509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.350903 16509 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:44043
I20260925 02:22:47.350986 16636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:44043 every 8 connection(s)
I20260925 02:22:47.351362 16509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:47.354055 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16509
I20260925 02:22:47.354162 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:47.354369 11764 external_mini_cluster.cc:1428] Setting key 5fcf2316d27c2edb209e8e5336ac8a1b
I20260925 02:22:47.354664 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.352160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33239 (local address 127.11.125.62:36745)
0925 02:22:47.352218 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.352220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.352451 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:22:47.353004 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.353005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.353011 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.353064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.354351 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.354352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.354352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.354356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.354373 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.354377 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.354452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.354550 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.354569 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:22:47.354872 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.352158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36745 (local address 127.11.125.1:33239)
0925 02:22:47.352340 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.352375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.352850 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.353121 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.353124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.353132 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.354172 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.354172 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.354213 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.354252 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.354454 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.354459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.354463 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.354548 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.354615 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 02:22:47.355351 16637 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36745
I20260925 02:22:47.355453 16637 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:47.355463 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--builtin_ntp_servers=127.11.125.20:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:22:47.355613 16637 heartbeater.cc:511] Master 127.11.125.62:36745 requested a full tablet report, sending...
I20260925 02:22:47.356204 16448 ts_manager.cc:194] Registered new tserver with Master: 47cddb96426c40919d358f159899c0e8 (127.11.125.1:44043)
W20260925 02:22:47.437157 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:47.437359 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:47.437382 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:47.438476 16640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:47.438508 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:47.438532 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:47.439777 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:47.439965 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:47.440291 16640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:47.445631 16646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.445590 16645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.445643 16640 server_base.cc:1034] running on GCE node
W20260925 02:22:47.445572 16648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.446087 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:47.446338 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:47.447502 16640 hybrid_clock.cc:648] HybridClock initialized: now 1790302967447488 us; error 27 us; skew 500 ppm
I20260925 02:22:47.448730 16640 webserver.cc:460] Webserver started at http://127.11.125.2:39715/ using document root <none> and password file <none>
I20260925 02:22:47.449038 16640 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:47.449090 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:47.449206 16640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:47.450470 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "175107485e5c4133b5e0199d00e063de"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "90d310096095f2339258d48bfa9b3226"
server_key_iv: "f622774bb7a4e2fc720b6f4be9e952c2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.450858 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "175107485e5c4133b5e0199d00e063de"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "90d310096095f2339258d48bfa9b3226"
server_key_iv: "f622774bb7a4e2fc720b6f4be9e952c2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.452219 16640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:47.453084 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.453256 16640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:47.453370 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "175107485e5c4133b5e0199d00e063de"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "90d310096095f2339258d48bfa9b3226"
server_key_iv: "f622774bb7a4e2fc720b6f4be9e952c2"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.453462 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:47.456781 16640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:47.457079 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:47.457230 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:47.457453 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:47.457774 16640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:47.457829 16640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.457870 16640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:47.457895 16640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.463738 16640 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:42019
I20260925 02:22:47.463781 16768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:42019 every 8 connection(s)
I20260925 02:22:47.464102 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:47.466874 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.464837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:43173 (local address 127.11.125.62:36745)
0925 02:22:47.464937 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.464940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.465145 (+ 205us) server_negotiation.cc:407] Connection header received
0925 02:22:47.465582 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.465584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.465589 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.465642 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.466674 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.466675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.466675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.466679 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.466694 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.466699 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.466745 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.466809 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.466821 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:22:47.467070 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.464788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36745 (local address 127.11.125.2:43173)
0925 02:22:47.465028 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.465062 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.465500 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.465692 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.465695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.465703 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.466542 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.466543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.466568 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.466594 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.466742 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.466744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.466747 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.466802 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.466848 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":128,"threads_started":1}
I20260925 02:22:47.467573 16769 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36745
I20260925 02:22:47.467666 16769 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:47.467794 16769 heartbeater.cc:511] Master 127.11.125.62:36745 requested a full tablet report, sending...
I20260925 02:22:47.468117 16448 ts_manager.cc:194] Registered new tserver with Master: 175107485e5c4133b5e0199d00e063de (127.11.125.2:42019)
I20260925 02:22:47.470571 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16640
I20260925 02:22:47.470674 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:47.470871 11764 external_mini_cluster.cc:1428] Setting key baf93a234abfd819b872fea1d0b1180c
I20260925 02:22:47.471803 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--builtin_ntp_servers=127.11.125.20:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:22:47.553723 16773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:47.553949 16773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:47.553973 16773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:47.555114 16773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:47.555151 16773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:47.555176 16773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:47.556661 16773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:47.556922 16773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:47.557221 16773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:47.562209 16781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.562209 16779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.562215 16778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.562494 16773 server_base.cc:1034] running on GCE node
I20260925 02:22:47.562740 16773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:47.562995 16773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:47.564155 16773 hybrid_clock.cc:648] HybridClock initialized: now 1790302967564142 us; error 42 us; skew 500 ppm
I20260925 02:22:47.565439 16773 webserver.cc:460] Webserver started at http://127.11.125.3:46819/ using document root <none> and password file <none>
I20260925 02:22:47.565696 16773 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:47.565778 16773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:47.565954 16773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:47.567006 16773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "bdc5c726a3ac41868892b59e6cad2105"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "9250c5a72fb0068d75b4fe18c41640f2"
server_key_iv: "1d622a1b310ec27dae16f1e18a19c90f"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.567368 16773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "bdc5c726a3ac41868892b59e6cad2105"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "9250c5a72fb0068d75b4fe18c41640f2"
server_key_iv: "1d622a1b310ec27dae16f1e18a19c90f"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.568674 16773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:22:47.569515 16787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.569793 16773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:47.569896 16773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "bdc5c726a3ac41868892b59e6cad2105"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "9250c5a72fb0068d75b4fe18c41640f2"
server_key_iv: "1d622a1b310ec27dae16f1e18a19c90f"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.569984 16773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:47.573333 16773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:47.573601 16773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:47.573743 16773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:47.573959 16773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:47.574260 16773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:47.574314 16773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.574359 16773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:47.574386 16773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.580283 16773 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:33681
I20260925 02:22:47.580328 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:33681 every 8 connection(s)
I20260925 02:22:47.580633 16773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:47.583406 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.581358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:44147 (local address 127.11.125.62:36745)
0925 02:22:47.581426 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.581428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.581634 (+ 206us) server_negotiation.cc:407] Connection header received
0925 02:22:47.582088 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.582089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.582096 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.582146 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.583205 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.583206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.583206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.583210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.583224 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.583228 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.583275 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.583340 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.583353 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:22:47.583576 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.581304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36745 (local address 127.11.125.3:44147)
0925 02:22:47.581524 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.581558 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.581999 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.582198 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.582201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.582209 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.583065 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.583066 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.583092 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.583119 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.583281 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.583284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.583286 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.583336 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.583382 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260925 02:22:47.584075 16903 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36745
I20260925 02:22:47.584152 16903 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:47.584259 16903 heartbeater.cc:511] Master 127.11.125.62:36745 requested a full tablet report, sending...
I20260925 02:22:47.584549 16448 ts_manager.cc:194] Registered new tserver with Master: bdc5c726a3ac41868892b59e6cad2105 (127.11.125.3:33681)
I20260925 02:22:47.586347 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16773
I20260925 02:22:47.586427 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:47.586549 11764 external_mini_cluster.cc:1428] Setting key b87aef8d059a2ca75f9ed432ee3c6ad8
I20260925 02:22:47.587286 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:22:47.589704 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.588562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.11.125.62:36745)
0925 02:22:47.588639 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.588640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.588748 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:22:47.588825 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.588826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.588832 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.588888 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.589485 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.589486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.589486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.589491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.589506 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.589511 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.589555 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.589617 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.589633 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:47.591192 16448 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:47.592813 16448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:47.596997 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.595002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:42019 (local address 127.0.0.1:43846)
0925 02:22:47.595580 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.595589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.595632 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.596180 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.596181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.596183 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.596510 (+ 327us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.596511 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.596516 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.596525 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.596826 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.596828 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.596828 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.596859 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.596873 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":90,"threads_started":1}
I20260925 02:22:47.597028 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.594995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:33681 (local address 127.0.0.1:39412)
0925 02:22:47.595164 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.595202 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.595270 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.595902 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.595909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.595917 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.596458 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.596458 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.596475 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.596502 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.596796 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.596803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.596806 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.596888 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.596905 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260925 02:22:47.597275 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.594995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44043 (local address 127.0.0.1:41930)
0925 02:22:47.595438 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.595447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.595493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.596351 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.596351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.596353 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.596690 (+ 337us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.596690 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.596696 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.596708 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.597002 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.597003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.597003 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.597195 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.597212 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260925 02:22:47.597378 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.595240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.11.125.2:42019)
0925 02:22:47.595407 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.595418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.595677 (+ 259us) server_negotiation.cc:407] Connection header received
0925 02:22:47.595707 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.595709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.595720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.595800 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.596652 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.596666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.596666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.596672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.596710 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.596746 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.597222 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.597247 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.597264 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 02:22:47.597371 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.595089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.11.125.3:33681)
0925 02:22:47.595747 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.595751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.595770 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:22:47.595795 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.595797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.595805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.595886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.596607 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.596615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.596616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.596620 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.596658 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.596683 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.596756 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.596936 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.596953 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":79,"threads_started":1}
I20260925 02:22:47.597676 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.595248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.11.125.1:44043)
0925 02:22:47.595410 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.595415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.595487 (+ 72us) server_negotiation.cc:407] Connection header received
0925 02:22:47.595555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.595558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.595569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.595652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.596810 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.596820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.596820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.596825 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.596859 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.596921 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.596976 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.597567 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.597585 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260925 02:22:47.598083 16836 tablet_service.cc:1467] Processing CreateTablet for tablet c63c8018efa04654aba7a4afd3b46d84 (DEFAULT_TABLE table=test-table [id=47a1cad8f8f4456c92d3eeb23b3522dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:47.598055 16702 tablet_service.cc:1467] Processing CreateTablet for tablet c63c8018efa04654aba7a4afd3b46d84 (DEFAULT_TABLE table=test-table [id=47a1cad8f8f4456c92d3eeb23b3522dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:47.598460 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63c8018efa04654aba7a4afd3b46d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:47.598476 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63c8018efa04654aba7a4afd3b46d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:47.598402 16571 tablet_service.cc:1467] Processing CreateTablet for tablet c63c8018efa04654aba7a4afd3b46d84 (DEFAULT_TABLE table=test-table [id=47a1cad8f8f4456c92d3eeb23b3522dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:47.598697 16571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63c8018efa04654aba7a4afd3b46d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:47.600930 16917 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Bootstrap starting.
I20260925 02:22:47.601639 16917 tablet_bootstrap.cc:654] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:47.601953 16917 log.cc:824] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:47.601977 16918 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Bootstrap starting.
I20260925 02:22:47.602596 16918 tablet_bootstrap.cc:654] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:47.602807 16917 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: No bootstrap required, opened a new log
I20260925 02:22:47.602885 16917 ts_tablet_manager.cc:1397] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:47.602921 16918 log.cc:824] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:47.603300 16919 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: Bootstrap starting.
I20260925 02:22:47.603724 16918 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: No bootstrap required, opened a new log
I20260925 02:22:47.603856 16918 ts_tablet_manager.cc:1397] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:47.604199 16919 tablet_bootstrap.cc:654] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:47.604529 16919 log.cc:824] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:47.605432 16919 tablet_bootstrap.cc:492] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: No bootstrap required, opened a new log
I20260925 02:22:47.605507 16919 ts_tablet_manager.cc:1397] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:47.605777 16917 raft_consensus.cc:348] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.605952 16917 raft_consensus.cc:374] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:47.605998 16917 raft_consensus.cc:729] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc5c726a3ac41868892b59e6cad2105, State: Initialized, Role: FOLLOWER
I20260925 02:22:47.606115 16917 consensus_queue.cc:260] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.606215 16918 raft_consensus.cc:348] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.606374 16918 raft_consensus.cc:374] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:47.606387 16917 ts_tablet_manager.cc:1428] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:22:47.606416 16918 raft_consensus.cc:729] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175107485e5c4133b5e0199d00e063de, State: Initialized, Role: FOLLOWER
I20260925 02:22:47.606506 16903 heartbeater.cc:503] Master 127.11.125.62:36745 was elected leader, sending a full tablet report...
I20260925 02:22:47.606516 16918 consensus_queue.cc:260] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.606786 16918 ts_tablet_manager.cc:1428] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:47.606906 16769 heartbeater.cc:503] Master 127.11.125.62:36745 was elected leader, sending a full tablet report...
I20260925 02:22:47.608157 16919 raft_consensus.cc:348] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.608283 16919 raft_consensus.cc:374] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:47.608304 16919 raft_consensus.cc:729] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cddb96426c40919d358f159899c0e8, State: Initialized, Role: FOLLOWER
I20260925 02:22:47.608377 16919 consensus_queue.cc:260] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.608606 16919 ts_tablet_manager.cc:1428] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:47.608662 16637 heartbeater.cc:503] Master 127.11.125.62:36745 was elected leader, sending a full tablet report...
W20260925 02:22:47.714661 16771 tablet.cc:2369] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:47.810465 16924 raft_consensus.cc:482] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:47.810684 16924 raft_consensus.cc:504] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.811015 16924 leader_election.cc:288] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47cddb96426c40919d358f159899c0e8 (127.11.125.1:44043), bdc5c726a3ac41868892b59e6cad2105 (127.11.125.3:33681)
I20260925 02:22:47.812561 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.811238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44043 (local address 127.11.125.2:50635)
0925 02:22:47.811336 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.811351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.811413 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.811652 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.811653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.811656 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.812175 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.812175 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.812183 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.812198 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.812387 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.812388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.812388 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.812460 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.812475 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:22:47.812590 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.811397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:50635 (local address 127.11.125.1:44043)
0925 02:22:47.811443 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.811445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.811459 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:22:47.811493 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.811494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.811502 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.811623 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.812292 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.812293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.812294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.812298 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.812319 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.812325 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.812389 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.812477 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.812497 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:22:47.812729 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.811238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:33681 (local address 127.11.125.2:53363)
0925 02:22:47.811510 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.811520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:47.811573 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:47.811760 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:47.811761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:47.811764 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:47.812343 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:22:47.812343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:47.812350 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:22:47.812362 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:47.812571 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:47.812572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:47.812572 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 02:22:47.812624 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:22:47.812636 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260925 02:22:47.812729 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.811359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:53363 (local address 127.11.125.3:33681)
0925 02:22:47.811411 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.811413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.811584 (+ 171us) server_negotiation.cc:407] Connection header received
0925 02:22:47.811644 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.811645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.811650 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.811710 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.812443 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.812444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.812445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.812450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.812468 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.812473 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.812544 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.812645 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.812661 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:22:47.813077 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63c8018efa04654aba7a4afd3b46d84" candidate_uuid: "175107485e5c4133b5e0199d00e063de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cddb96426c40919d358f159899c0e8" is_pre_election: true
I20260925 02:22:47.813230 16591 raft_consensus.cc:2454] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175107485e5c4133b5e0199d00e063de in term 0.
I20260925 02:22:47.813280 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63c8018efa04654aba7a4afd3b46d84" candidate_uuid: "175107485e5c4133b5e0199d00e063de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc5c726a3ac41868892b59e6cad2105" is_pre_election: true
I20260925 02:22:47.813455 16856 raft_consensus.cc:2454] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 175107485e5c4133b5e0199d00e063de in term 0.
I20260925 02:22:47.813493 16656 leader_election.cc:302] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175107485e5c4133b5e0199d00e063de, 47cddb96426c40919d358f159899c0e8; no voters:
I20260925 02:22:47.813602 16924 raft_consensus.cc:2781] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:47.813657 16924 raft_consensus.cc:482] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:47.813678 16924 raft_consensus.cc:3037] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:47.814401 16924 raft_consensus.cc:504] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.814554 16924 leader_election.cc:288] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [CANDIDATE]: Term 1 election: Requested vote from peers 47cddb96426c40919d358f159899c0e8 (127.11.125.1:44043), bdc5c726a3ac41868892b59e6cad2105 (127.11.125.3:33681)
I20260925 02:22:47.814668 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63c8018efa04654aba7a4afd3b46d84" candidate_uuid: "175107485e5c4133b5e0199d00e063de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cddb96426c40919d358f159899c0e8"
I20260925 02:22:47.814708 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63c8018efa04654aba7a4afd3b46d84" candidate_uuid: "175107485e5c4133b5e0199d00e063de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc5c726a3ac41868892b59e6cad2105"
I20260925 02:22:47.814778 16856 raft_consensus.cc:3037] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:47.814769 16591 raft_consensus.cc:3037] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:47.815653 16856 raft_consensus.cc:2454] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175107485e5c4133b5e0199d00e063de in term 1.
I20260925 02:22:47.815653 16591 raft_consensus.cc:2454] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 175107485e5c4133b5e0199d00e063de in term 1.
I20260925 02:22:47.815829 16656 leader_election.cc:302] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 175107485e5c4133b5e0199d00e063de, 47cddb96426c40919d358f159899c0e8; no voters:
I20260925 02:22:47.815964 16924 raft_consensus.cc:2781] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:47.816164 16924 raft_consensus.cc:686] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 1 LEADER]: Becoming Leader. State: Replica: 175107485e5c4133b5e0199d00e063de, State: Running, Role: LEADER
I20260925 02:22:47.816291 16924 consensus_queue.cc:237] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } }
I20260925 02:22:47.817212 16447 catalog_manager.cc:5495] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de reported cstate change: term changed from 0 to 1, leader changed from <none> to 175107485e5c4133b5e0199d00e063de (127.11.125.2). New cstate: current_term: 1 leader_uuid: "175107485e5c4133b5e0199d00e063de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "175107485e5c4133b5e0199d00e063de" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42019 } health_report { overall_health: HEALTHY } } }
I20260925 02:22:47.827455 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.826365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.11.125.2:42019)
0925 02:22:47.826436 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:47.826438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:47.826445 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:22:47.826481 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:47.826482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:47.826496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:47.826549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:47.827148 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:47.827150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:47.827150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:47.827170 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:47.827190 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:47.827195 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:47.827279 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:47.827340 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:22:47.827373 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:22:47.829737 16856 raft_consensus.cc:1261] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 1 FOLLOWER]: Refusing update from remote peer 175107485e5c4133b5e0199d00e063de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:47.829748 16591 raft_consensus.cc:1261] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 1 FOLLOWER]: Refusing update from remote peer 175107485e5c4133b5e0199d00e063de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:47.830075 16927 consensus_queue.cc:1035] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdc5c726a3ac41868892b59e6cad2105" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:47.830195 16924 consensus_queue.cc:1035] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [LEADER]: Connected to new peer: Peer: permanent_uuid: "47cddb96426c40919d358f159899c0e8" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:22:47.831184 16904 tablet.cc:2369] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:47.833355 16932 mvcc.cc:205] Tried to move back new op lower bound from 7333080956227940352 to 7333080956176117760. Current Snapshot: MvccSnapshot[applied={T|T < 7333080956227940352 or (T in {7333080956227940352})}]
I20260925 02:22:47.833478 16931 mvcc.cc:205] Tried to move back new op lower bound from 7333080956227940352 to 7333080956176117760. Current Snapshot: MvccSnapshot[applied={T|T < 7333080956227940352}]
I20260925 02:22:47.835467 16447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:47.835572 16447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:47.836346 16447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b11ba4d2fa4742cabab74ea031696ebb: Sending DeleteTablet for 3 replicas of tablet c63c8018efa04654aba7a4afd3b46d84
I20260925 02:22:47.836683 16571 tablet_service.cc:1514] Processing DeleteTablet for tablet c63c8018efa04654aba7a4afd3b46d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='slave'} at 127.0.0.1:41930
I20260925 02:22:47.836792 16702 tablet_service.cc:1514] Processing DeleteTablet for tablet c63c8018efa04654aba7a4afd3b46d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='slave'} at 127.0.0.1:43846
I20260925 02:22:47.836962 16836 tablet_service.cc:1514] Processing DeleteTablet for tablet c63c8018efa04654aba7a4afd3b46d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:47 UTC) from {username='slave'} at 127.0.0.1:39412
I20260925 02:22:47.837002 16939 tablet_replica.cc:330] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8: stopping tablet replica
I20260925 02:22:47.837098 16939 raft_consensus.cc:2229] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:47.837139 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16509
I20260925 02:22:47.837203 16939 raft_consensus.cc:2258] T c63c8018efa04654aba7a4afd3b46d84 P 47cddb96426c40919d358f159899c0e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:47.841509 16940 tablet_replica.cc:330] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: stopping tablet replica
I20260925 02:22:47.841627 16940 raft_consensus.cc:2229] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:47.841804 16940 raft_consensus.cc:2258] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:47.842284 16940 ts_tablet_manager.cc:1905] T c63c8018efa04654aba7a4afd3b46d84 P 175107485e5c4133b5e0199d00e063de: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:22:47.843657 16433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:44043 (error 108)
I20260925 02:22:47.843885 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16640
I20260925 02:22:47.844228 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.844095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44043 (local address 127.0.0.1:41938)
0925 02:22:47.844146 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.844189 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 02:22:47.844328 16433 catalog_manager.cc:4584] TS 47cddb96426c40919d358f159899c0e8 (127.11.125.1:44043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c63c8018efa04654aba7a4afd3b46d84: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44043: connect: Connection refused (error 111)
I20260925 02:22:47.844542 16941 tablet_replica.cc:330] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: stopping tablet replica
I20260925 02:22:47.844663 16941 raft_consensus.cc:2229] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:47.844789 16941 raft_consensus.cc:2258] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:47.845264 16941 ts_tablet_manager.cc:1905] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:47.846879 16941 ts_tablet_manager.cc:1918] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:47.846940 16941 log.cc:1196] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/c63c8018efa04654aba7a4afd3b46d84
I20260925 02:22:47.847198 16941 ts_tablet_manager.cc:1939] T c63c8018efa04654aba7a4afd3b46d84 P bdc5c726a3ac41868892b59e6cad2105: Deleting consensus metadata
I20260925 02:22:47.847597 16435 catalog_manager.cc:4841] TS bdc5c726a3ac41868892b59e6cad2105 (127.11.125.3:33681): tablet c63c8018efa04654aba7a4afd3b46d84 (table test-table [id=47a1cad8f8f4456c92d3eeb23b3522dd]) successfully deleted
I20260925 02:22:47.849596 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16773
I20260925 02:22:47.849633 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:47.849538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:42019 (local address 127.0.0.1:43870)
0925 02:22:47.849580 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:47.849607 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.2:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 02:22:47.855085 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16417
2026-09-25T02:22:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (785 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:22:47.875412 11764 mini_oidc.cc:152] Starting JWKS server
I20260925 02:22:47.876484 11764 webserver.cc:460] Webserver started at https://127.0.0.1:44703/ using document root <none> and password file <none>
I20260925 02:22:47.876608 11764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:22:47.876793 11764 webserver.cc:460] Webserver started at http://127.0.0.1:44735/ using document root <none> and password file <none>
2026-09-25T02:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:47Z Disabled control of system clock
I20260925 02:22:47.884582 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42107
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:42107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44703/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/testchainca.cert with env {}
W20260925 02:22:47.968225 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:47.968513 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:47.968554 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:47.969810 16948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:47.969893 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:47.969916 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:47.969950 16948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:47.969977 16948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:22:47.970005 16948 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:22:47.970029 16948 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44703/jwks.json
I20260925 02:22:47.971884 16948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:42107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44703/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:47.972179 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:47.972409 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:47.977428 16953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.977406 16954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:47.977406 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:47.977802 16948 server_base.cc:1034] running on GCE node
I20260925 02:22:47.978062 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:47.978359 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:47.979503 16948 hybrid_clock.cc:648] HybridClock initialized: now 1790302967979487 us; error 31 us; skew 500 ppm
I20260925 02:22:47.986706 16948 webserver.cc:460] Webserver started at http://127.11.125.62:44847/ using document root <none> and password file <none>
I20260925 02:22:47.986989 16948 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:47.987040 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:47.987213 16948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:47.988400 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "f9080e1324ac4d178dfabc55e87a95b3"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "662d7b1f60dd762135e6881aa7b8c074"
server_key_iv: "4073b6db955876614d9e3c61c174cab8"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.988792 16948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "f9080e1324ac4d178dfabc55e87a95b3"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "662d7b1f60dd762135e6881aa7b8c074"
server_key_iv: "4073b6db955876614d9e3c61c174cab8"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.990227 16948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:47.991081 16963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:47.991273 16948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:47.991393 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "f9080e1324ac4d178dfabc55e87a95b3"
format_stamp: "Formatted at 2026-09-25 02:22:47 on dist-test-slave-hj7s"
server_key: "662d7b1f60dd762135e6881aa7b8c074"
server_key_iv: "4073b6db955876614d9e3c61c174cab8"
server_key_version: "encryptionkey@0"
I20260925 02:22:47.991477 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:48.016474 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:48.016846 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:48.017026 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:48.021322 16948 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:42107
I20260925 02:22:48.021378 17015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:42107 every 8 connection(s)
I20260925 02:22:48.021795 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:48.022840 17016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:48.025380 17016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: Bootstrap starting.
I20260925 02:22:48.026013 17016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:48.026377 17016 log.cc:824] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:48.027148 17016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: No bootstrap required, opened a new log
I20260925 02:22:48.028800 17016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } }
I20260925 02:22:48.029013 17016 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:48.029043 17016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9080e1324ac4d178dfabc55e87a95b3, State: Initialized, Role: FOLLOWER
I20260925 02:22:48.029220 17016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } }
I20260925 02:22:48.029295 17016 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:48.029342 17016 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:48.029392 17016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:48.029471 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 16948
I20260925 02:22:48.029595 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:48.029848 11764 external_mini_cluster.cc:1428] Setting key 4c0751354af75c0b1fcca2308d92ea5e
I20260925 02:22:48.030133 17016 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } }
I20260925 02:22:48.030259 17016 leader_election.cc:302] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9080e1324ac4d178dfabc55e87a95b3; no voters:
I20260925 02:22:48.030503 17016 leader_election.cc:288] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:48.030560 17020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:48.030719 17020 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [term 1 LEADER]: Becoming Leader. State: Replica: f9080e1324ac4d178dfabc55e87a95b3, State: Running, Role: LEADER
I20260925 02:22:48.030844 17016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:48.030866 17020 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } }
I20260925 02:22:48.031415 17022 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9080e1324ac4d178dfabc55e87a95b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } } }
I20260925 02:22:48.031502 17022 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.031514 17023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9080e1324ac4d178dfabc55e87a95b3. Latest consensus state: current_term: 1 leader_uuid: "f9080e1324ac4d178dfabc55e87a95b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9080e1324ac4d178dfabc55e87a95b3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42107 } } }
I20260925 02:22:48.031607 17023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:48.032850 17037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:48.032923 17037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:48.033082 17029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:48.033269 17029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:48.035733 17029 catalog_manager.cc:1347] Generated new cluster ID: 3d61676a6c22499c848d046eeab6e329
I20260925 02:22:48.035864 17029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:48.036131 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.030412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.11.125.62:42107)
0925 02:22:48.030683 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.030694 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.030730 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:22:48.030864 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.030871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.030945 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.031371 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:48.031487 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.032862 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.033985 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.034429 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.034569 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.035352 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:48.035356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:48.035356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:48.035535 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:48.035594 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:48.035610 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:48.035676 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:48.035813 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:22:48.035893 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260925 02:22:48.037165 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:22:48.054762 17029 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:48.055523 17029 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:48.061424 17029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9080e1324ac4d178dfabc55e87a95b3: Generated new TSK 0
I20260925 02:22:48.061658 17029 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:22:48.087661 17046 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:22:48.087785 17046 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:22:48.088119 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.071708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.11.125.62:42107)
0925 02:22:48.071761 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.071763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.071857 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:22:48.071926 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.071927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.071940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.072035 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:22:48.072164 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.072990 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.073746 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.074056 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.074131 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.074357 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:22:48.087835 (+ 13478us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:22:48.088044 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 02:22:48.088177 17021 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:22:48.088513 17045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:48.071613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42107 (local address 127.0.0.1:59952)
0925 02:22:48.071780 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:48.071790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:48.071850 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:48.072035 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:48.072036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:48.072038 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:22:48.072097 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.072099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.073067 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.073067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.073657 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.073659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.074147 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.074148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.074193 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.074213 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:22:48.088022 (+ 13809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:22:48.088139 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.125.62:42107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.125.62:42107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:22:48.089038 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 16948
2026-09-25T02:22:48Z chronyd exiting
I20260925 02:22:48.125926 11764 test_util.cc:175] -----------------------------------------------
I20260925 02:22:48.126009 11764 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790302942477976-11764-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:22:48.126878 11764 mini_oidc.cc:152] Starting JWKS server
I20260925 02:22:48.127573 11764 webserver.cc:460] Webserver started at https://127.0.0.1:38467/ using document root <none> and password file <none>
I20260925 02:22:48.127615 11764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:22:48.127807 11764 webserver.cc:460] Webserver started at http://127.0.0.1:39133/ using document root <none> and password file <none>
2026-09-25T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:48Z Disabled control of system clock
I20260925 02:22:48.136124 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38225
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:38225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38467/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/testchainca.cert with env {}
W20260925 02:22:48.217571 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:48.217787 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:48.217809 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:48.219019 17052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:48.219062 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:48.219075 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:48.219089 17052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:48.219141 17052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:22:48.219168 17052 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:22:48.219193 17052 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38467/jwks.json
I20260925 02:22:48.220722 17052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:38225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38467/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:48.221011 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:48.221235 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:48.226221 17058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:48.226209 17060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:48.226209 17057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:48.226405 17052 server_base.cc:1034] running on GCE node
I20260925 02:22:48.226643 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:48.226945 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:48.228111 17052 hybrid_clock.cc:648] HybridClock initialized: now 1790302968228097 us; error 46 us; skew 500 ppm
I20260925 02:22:48.229353 17052 webserver.cc:460] Webserver started at http://127.11.125.62:39663/ using document root <none> and password file <none>
I20260925 02:22:48.229588 17052 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:48.229656 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:48.229815 17052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:48.230948 17052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "d3a4373984d249d0af639c662def716e"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "7862cfbe4ae12cb815b46d96b73c861a"
server_key_iv: "44d7f90b5bb329588fb97db13dc8a986"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.231333 17052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "d3a4373984d249d0af639c662def716e"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "7862cfbe4ae12cb815b46d96b73c861a"
server_key_iv: "44d7f90b5bb329588fb97db13dc8a986"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.232645 17052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:48.233466 17066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:48.233639 17052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:48.233744 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "d3a4373984d249d0af639c662def716e"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "7862cfbe4ae12cb815b46d96b73c861a"
server_key_iv: "44d7f90b5bb329588fb97db13dc8a986"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.233829 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:48.243534 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:48.243877 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:48.244032 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:48.248032 17052 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:38225
I20260925 02:22:48.248086 17119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:38225 every 8 connection(s)
I20260925 02:22:48.248466 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:48.249229 17120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:48.250382 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17052
I20260925 02:22:48.250494 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:48.250682 11764 external_mini_cluster.cc:1428] Setting key 5248e59460cb06923f9e47bc9d16ac30
I20260925 02:22:48.252784 17120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e: Bootstrap starting.
I20260925 02:22:48.253535 17120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:48.253862 17120 log.cc:824] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:48.254582 17120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e: No bootstrap required, opened a new log
I20260925 02:22:48.256261 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.251250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.11.125.62:38225)
0925 02:22:48.251462 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.251475 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.251501 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:22:48.251915 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.251922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.251982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.252117 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:48.252337 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.253311 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.254370 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.254816 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.254908 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.255593 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:48.255597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:48.255597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:48.255759 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:48.255795 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:48.255805 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:48.255857 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:48.255969 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:22:48.256068 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260925 02:22:48.256834 17120 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } }
I20260925 02:22:48.257071 17120 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:48.257128 17120 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a4373984d249d0af639c662def716e, State: Initialized, Role: FOLLOWER
I20260925 02:22:48.257293 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:22:48.257273 17120 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } }
I20260925 02:22:48.257355 17120 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:48.257383 17120 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:48.257431 17120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:48.258108 17120 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } }
I20260925 02:22:48.258229 17120 leader_election.cc:302] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a4373984d249d0af639c662def716e; no voters:
I20260925 02:22:48.258402 17120 leader_election.cc:288] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:48.258451 17125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:48.258571 17125 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [term 1 LEADER]: Becoming Leader. State: Replica: d3a4373984d249d0af639c662def716e, State: Running, Role: LEADER
I20260925 02:22:48.258659 17120 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:48.258659 17125 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } }
I20260925 02:22:48.259025 17125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a4373984d249d0af639c662def716e. Latest consensus state: current_term: 1 leader_uuid: "d3a4373984d249d0af639c662def716e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } } }
I20260925 02:22:48.259037 17127 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a4373984d249d0af639c662def716e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a4373984d249d0af639c662def716e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38225 } } }
I20260925 02:22:48.259126 17125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.259140 17127 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.259368 17131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:48.259898 17131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:48.261445 17131 catalog_manager.cc:1347] Generated new cluster ID: b8c2d2f9abc24668b8b682ba47df7c31
I20260925 02:22:48.261497 17131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:48.268260 17131 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:48.269024 17131 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:48.279349 17131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a4373984d249d0af639c662def716e: Generated new TSK 0
I20260925 02:22:48.279564 17131 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:22:48.300783 17150 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:22:48.300882 17150 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:22:48.301333 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.286992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.11.125.62:38225)
0925 02:22:48.287061 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.287064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.287176 (+ 112us) server_negotiation.cc:407] Connection header received
0925 02:22:48.287257 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.287257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.287271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.287360 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:22:48.287485 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.288003 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.288806 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.289089 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.289165 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.289369 (+ 204us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:22:48.301051 (+ 11682us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:22:48.301247 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:22:48.301401 17123 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 02:22:48.301424 17149 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:48.286929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38225 (local address 127.0.0.1:38066)
0925 02:22:48.287103 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:48.287113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:48.287193 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:48.287362 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:48.287363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:48.287365 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:22:48.287418 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.287421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.288098 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.288099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.288709 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.288711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.289169 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.289170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.289234 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.289241 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:22:48.301198 (+ 11957us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:22:48.301312 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.125.62:38225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38467/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260925 02:22:48.301795 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17052
2026-09-25T02:22:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:22:48.339769 11764 mini_oidc.cc:152] Starting JWKS server
I20260925 02:22:48.340440 11764 webserver.cc:460] Webserver started at https://127.0.0.1:33849/ using document root <none> and password file <none>
I20260925 02:22:48.340480 11764 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:22:48.340618 11764 webserver.cc:460] Webserver started at http://127.0.0.1:41029/ using document root <none> and password file <none>
2026-09-25T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:48Z Disabled control of system clock
I20260925 02:22:48.348249 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35515
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:35515
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:22:48.428442 17156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:48.428696 17156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:48.428732 17156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:48.429947 17156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:48.430022 17156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:48.430043 17156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:48.430064 17156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:48.430096 17156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:22:48.430120 17156 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:22:48.430148 17156 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33849/jwks.json
I20260925 02:22:48.431577 17156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:35515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:48.431870 17156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:48.432165 17156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:48.437104 17162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:48.437191 17161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:48.437237 17156 server_base.cc:1034] running on GCE node
W20260925 02:22:48.437095 17164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:48.437604 17156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:48.437903 17156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:48.439067 17156 hybrid_clock.cc:648] HybridClock initialized: now 1790302968439051 us; error 45 us; skew 500 ppm
I20260925 02:22:48.440282 17156 webserver.cc:460] Webserver started at http://127.11.125.62:38445/ using document root <none> and password file <none>
I20260925 02:22:48.440528 17156 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:48.440596 17156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:48.440744 17156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:48.441929 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "5b6b86ea7f844e70b0eb768a5ec90394"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "e8d73f35a0da3bf02300a0c647a263ba"
server_key_iv: "e0eb6a09cf8d6e41d65d13e0d7aced9b"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.442301 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "5b6b86ea7f844e70b0eb768a5ec90394"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "e8d73f35a0da3bf02300a0c647a263ba"
server_key_iv: "e0eb6a09cf8d6e41d65d13e0d7aced9b"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.443696 17156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:48.444432 17170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:48.444607 17156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:48.444705 17156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "5b6b86ea7f844e70b0eb768a5ec90394"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "e8d73f35a0da3bf02300a0c647a263ba"
server_key_iv: "e0eb6a09cf8d6e41d65d13e0d7aced9b"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.444796 17156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:48.456418 17156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:48.456719 17156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:48.456879 17156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:48.460836 17156 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:35515
I20260925 02:22:48.460889 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:35515 every 8 connection(s)
I20260925 02:22:48.461279 17156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:48.462218 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17156
I20260925 02:22:48.462317 17223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:48.462337 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:48.462523 11764 external_mini_cluster.cc:1428] Setting key c2fd151f8af011da092a8aec6d884990
I20260925 02:22:48.466104 17223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394: Bootstrap starting.
I20260925 02:22:48.466784 17223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:48.467092 17223 log.cc:824] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:48.467751 17223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394: No bootstrap required, opened a new log
I20260925 02:22:48.468698 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.463106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.11.125.62:35515)
0925 02:22:48.463325 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.463332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.463358 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:22:48.464035 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.464042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.464125 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.464245 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:48.464542 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.465567 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.466638 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.467125 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.467270 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.468044 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:48.468048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:48.468049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:48.468239 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:48.468272 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:48.468284 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:48.468361 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:48.468444 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:22:48.468529 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260925 02:22:48.469847 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:22:48.469863 17223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } }
I20260925 02:22:48.470045 17223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:48.470094 17223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6b86ea7f844e70b0eb768a5ec90394, State: Initialized, Role: FOLLOWER
I20260925 02:22:48.470218 17223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } }
I20260925 02:22:48.470311 17223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:48.470357 17223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:48.470407 17223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:48.471036 17223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } }
I20260925 02:22:48.471144 17223 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6b86ea7f844e70b0eb768a5ec90394; no voters:
I20260925 02:22:48.471309 17223 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:48.471347 17228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:48.471590 17228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6b86ea7f844e70b0eb768a5ec90394, State: Running, Role: LEADER
I20260925 02:22:48.471669 17223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:48.471700 17228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } }
I20260925 02:22:48.472050 17230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } } }
I20260925 02:22:48.472056 17229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6b86ea7f844e70b0eb768a5ec90394. Latest consensus state: current_term: 1 leader_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6b86ea7f844e70b0eb768a5ec90394" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35515 } } }
I20260925 02:22:48.472226 17230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.472245 17229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.472589 17237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:48.473133 17237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:48.474844 17237 catalog_manager.cc:1347] Generated new cluster ID: d354fb26a2b84c2abe77a5b32406ed33
I20260925 02:22:48.474915 17237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:48.491026 17237 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:48.491598 17237 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:48.510849 17237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6b86ea7f844e70b0eb768a5ec90394: Generated new TSK 0
I20260925 02:22:48.511121 17237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:48.530892 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.527344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.11.125.62:35515)
0925 02:22:48.527411 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.527414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.527514 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:22:48.527590 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.527591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.527603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.527663 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:22:48.527815 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.528284 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.529428 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.529604 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.529673 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.529867 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:22:48.530656 (+ 789us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:22:48.530811 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:22:48.530965 17226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:22:48.530975 17253 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:48.527272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:35515 (local address 127.0.0.1:32996)
0925 02:22:48.527439 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:48.527451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:48.527520 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:48.527683 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:48.527684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:48.527686 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:22:48.527751 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.527754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.528367 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.528368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.529353 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.529356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.529701 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.529702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.529763 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.529769 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:22:48.530771 (+ 1002us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:22:48.530865 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.125.62:35515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":114,"threads_started":1}
I20260925 02:22:48.531131 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17156
E20260925 02:22:48.542219 17254 webserver.cc:533] Webserver: sslize failed:
E20260925 02:22:48.542287 17254 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T02:22:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:48Z Disabled control of system clock
I20260925 02:22:48.577096 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39525
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:39525
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/testchainca.cert with env {}
W20260925 02:22:48.657608 17258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:48.657816 17258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:48.657840 17258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:48.658963 17258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:48.658996 17258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:48.659009 17258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:48.659022 17258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:48.659034 17258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:22:48.659045 17258 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:22:48.659057 17258 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:22:48.660481 17258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:39525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:48.660729 17258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:48.660955 17258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:48.665962 17264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:48.665997 17263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:48.666029 17258 server_base.cc:1034] running on GCE node
W20260925 02:22:48.665962 17266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:48.666375 17258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:48.666692 17258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:48.667863 17258 hybrid_clock.cc:648] HybridClock initialized: now 1790302968667859 us; error 41 us; skew 500 ppm
I20260925 02:22:48.669142 17258 webserver.cc:460] Webserver started at http://127.11.125.62:45679/ using document root <none> and password file <none>
I20260925 02:22:48.669389 17258 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:48.669459 17258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:48.669610 17258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:48.670689 17258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "79d10ed769aa4929bd5f02d2a8e31644"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "272fb010c26356de5f59b5dda8f70ec6"
server_key_iv: "a1cd54c8667c2b1322d5618e0370cf78"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.671066 17258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "79d10ed769aa4929bd5f02d2a8e31644"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "272fb010c26356de5f59b5dda8f70ec6"
server_key_iv: "a1cd54c8667c2b1322d5618e0370cf78"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.672509 17258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:48.673283 17272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:48.673481 17258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:22:48.673576 17258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "79d10ed769aa4929bd5f02d2a8e31644"
format_stamp: "Formatted at 2026-09-25 02:22:48 on dist-test-slave-hj7s"
server_key: "272fb010c26356de5f59b5dda8f70ec6"
server_key_iv: "a1cd54c8667c2b1322d5618e0370cf78"
server_key_version: "encryptionkey@0"
I20260925 02:22:48.673631 17258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:48.688195 17258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:48.688452 17258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:48.688552 17258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:48.692709 17258 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:39525
I20260925 02:22:48.692741 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:39525 every 8 connection(s)
I20260925 02:22:48.693152 17258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:48.694154 17325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:48.696462 17325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644: Bootstrap starting.
I20260925 02:22:48.697131 17325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:48.697430 17325 log.cc:824] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:48.698093 17325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644: No bootstrap required, opened a new log
I20260925 02:22:48.699700 17325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } }
I20260925 02:22:48.699851 17325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:48.699873 17325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d10ed769aa4929bd5f02d2a8e31644, State: Initialized, Role: FOLLOWER
I20260925 02:22:48.700063 17325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } }
I20260925 02:22:48.700155 17325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:48.700178 17325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:48.700246 17325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:48.700779 17325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } }
I20260925 02:22:48.700892 17325 leader_election.cc:302] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d10ed769aa4929bd5f02d2a8e31644; no voters:
I20260925 02:22:48.701102 17325 leader_election.cc:288] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:48.701156 17328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:48.701344 17328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [term 1 LEADER]: Becoming Leader. State: Replica: 79d10ed769aa4929bd5f02d2a8e31644, State: Running, Role: LEADER
I20260925 02:22:48.701381 17325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:48.701399 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17258
I20260925 02:22:48.701455 17328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } }
I20260925 02:22:48.701509 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:48.701730 11764 external_mini_cluster.cc:1428] Setting key 0d059a3ae8497cf475739ff782dd24ec
I20260925 02:22:48.701849 17329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d10ed769aa4929bd5f02d2a8e31644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } } }
I20260925 02:22:48.701927 17329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.702389 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d10ed769aa4929bd5f02d2a8e31644. Latest consensus state: current_term: 1 leader_uuid: "79d10ed769aa4929bd5f02d2a8e31644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d10ed769aa4929bd5f02d2a8e31644" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39525 } } }
I20260925 02:22:48.702440 17335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:48.703039 17335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:48.703529 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:48.704954 17335 catalog_manager.cc:1347] Generated new cluster ID: b876bc545bff4494849517b894773524
I20260925 02:22:48.705013 17335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:48.707783 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.702386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.11.125.62:39525)
0925 02:22:48.702572 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.702581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.702660 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:22:48.702875 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.702882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.702964 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.703118 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:48.703358 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.704577 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.705688 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.706272 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.706383 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.707082 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:48.707085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:48.707086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:22:48.707293 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:48.707332 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:22:48.707346 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:48.707408 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:48.707513 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:22:48.707585 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260925 02:22:48.708678 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:22:48.725334 17335 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:48.726104 17335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:48.731869 17335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d10ed769aa4929bd5f02d2a8e31644: Generated new TSK 0
I20260925 02:22:48.732082 17335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:48.760524 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:48.742040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.11.125.62:39525)
0925 02:22:48.742106 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:48.742109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:48.742234 (+ 125us) server_negotiation.cc:407] Connection header received
0925 02:22:48.742299 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:48.742300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:48.742321 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:48.742380 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:22:48.742537 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.743015 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.744158 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.744339 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.744384 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.744576 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:22:48.760298 (+ 15722us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:22:48.760461 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40800: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 02:22:48.760609 17341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40800: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:22:48.760820 17354 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:48.741985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39525 (local address 127.0.0.1:40800)
0925 02:22:48.742155 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:48.742167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:48.742237 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:48.742404 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:48.742405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:48.742407 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:22:48.742473 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.742475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.743092 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.743093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.744068 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:48.744071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:48.744415 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:48.744416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:48.744478 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:48.744484 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:22:48.760447 (+ 15963us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:22:48.760594 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.125.62:39525: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260925 02:22:48.761248 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17258
2026-09-25T02:22:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (209 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:48 dist-test-slave-hj7s krb5kdc[17366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:48 dist-test-slave-hj7s krb5kdc[17366](info): set up 2 sockets
Sep 25 02:22:48 dist-test-slave-hj7s krb5kdc[17366](info): commencing operation
krb5kdc: starting...
W20260925 02:22:50.003518 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:22:50.124255 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35619
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:35619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:50.207193 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:50.207474 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:50.207515 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:50.208747 17379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:50.208830 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:50.208853 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:50.208875 17379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:50.208927 17379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:50.210458 17379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:35619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:50.210764 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:50.211023 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:50.216177 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.216177 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.216172 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:50.216351 17379 server_base.cc:1034] running on GCE node
I20260925 02:22:50.216632 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:50.217012 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:50.218211 17379 hybrid_clock.cc:648] HybridClock initialized: now 1790302970218186 us; error 56 us; skew 500 ppm
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.220835 17379 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:22:50.221381 17379 webserver.cc:460] Webserver started at http://127.11.125.62:41417/ using document root <none> and password file <none>
I20260925 02:22:50.221634 17379 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:50.221709 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:50.221863 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:50.222934 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "0460c6d36cfa1b1e22e517b4cb8806b0"
server_key_iv: "06b9af508c75029ea5144dc9b02fd555"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.223300 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "0460c6d36cfa1b1e22e517b4cb8806b0"
server_key_iv: "06b9af508c75029ea5144dc9b02fd555"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.224611 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:50.225418 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.225589 17379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.225696 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "0460c6d36cfa1b1e22e517b4cb8806b0"
server_key_iv: "06b9af508c75029ea5144dc9b02fd555"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.225781 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:50.238296 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:50.240358 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:50.240540 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:50.244678 17379 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:35619
I20260925 02:22:50.244720 17447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:35619 every 8 connection(s)
I20260925 02:22:50.245132 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:50.246111 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:50.248485 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: Bootstrap starting.
I20260925 02:22:50.249151 17448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:50.249361 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17379
I20260925 02:22:50.249465 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:50.249500 17448 log.cc:824] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:50.249687 11764 external_mini_cluster.cc:1428] Setting key 2e4aecf946d0313408cf3d9ee1a22c9a
I20260925 02:22:50.250402 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: No bootstrap required, opened a new log
I20260925 02:22:50.253238 17448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } }
I20260925 02:22:50.253425 17448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:50.253461 17448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9bb69f3c9b49db94e1acf8a6a4f71f, State: Initialized, Role: FOLLOWER
I20260925 02:22:50.253597 17448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } }
I20260925 02:22:50.253666 17448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:50.253705 17448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:50.253747 17448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:50.254361 17448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } }
I20260925 02:22:50.254487 17448 leader_election.cc:302] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c9bb69f3c9b49db94e1acf8a6a4f71f; no voters:
I20260925 02:22:50.254712 17448 leader_election.cc:288] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:50.254732 17453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:50.254853 17453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [term 1 LEADER]: Becoming Leader. State: Replica: 7c9bb69f3c9b49db94e1acf8a6a4f71f, State: Running, Role: LEADER
I20260925 02:22:50.254935 17453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } }
I20260925 02:22:50.254988 17448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:50.256525 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c9bb69f3c9b49db94e1acf8a6a4f71f. Latest consensus state: current_term: 1 leader_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } } }
I20260925 02:22:50.256625 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9bb69f3c9b49db94e1acf8a6a4f71f" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35619 } } }
I20260925 02:22:50.256732 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:50.256696 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:50.256808 17467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:50.256944 17467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:50.257103 17469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:50.257261 17469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:50.259145 17469 catalog_manager.cc:1347] Generated new cluster ID: 984a3eae64af4728a1fc138f63adafdb
I20260925 02:22:50.259202 17469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:50.261212 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.250265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.11.125.62:35619)
0925 02:22:50.250434 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.250444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.250497 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:22:50.250954 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.250960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.251010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.251124 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.251955 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.252850 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.253975 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.254411 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.254582 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.257561 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.257565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.257565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.257577 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.259628 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.260000 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.260004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.260004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.260094 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.260307 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.260308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.260309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.260549 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.260725 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.260831 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.260996 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260925 02:22:50.271554 17469 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:50.272269 17469 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:50.280069 17469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c9bb69f3c9b49db94e1acf8a6a4f71f: Generated new TSK 0
I20260925 02:22:50.280256 17469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:22:50.307370 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:50.390267 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:50.390480 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:50.390504 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:50.391610 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:50.391647 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:50.391724 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:50.393229 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:50.393416 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:50.393586 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:50.398808 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.398797 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.398869 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:50.399070 17474 server_base.cc:1034] running on GCE node
I20260925 02:22:50.399286 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:50.399487 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:50.400625 17474 hybrid_clock.cc:648] HybridClock initialized: now 1790302970400613 us; error 34 us; skew 500 ppm
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.403149 17474 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:22:50.403644 17474 webserver.cc:460] Webserver started at http://127.11.125.1:36489/ using document root <none> and password file <none>
I20260925 02:22:50.403884 17474 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:50.403968 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:50.404129 17474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:50.405325 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "038a51c29e5f47af8fe782faf7f6b6c8"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "7fe824dac7098821f9dee09f67626d67"
server_key_iv: "350a49b0535e9f3fffa39a5d47e5e333"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.405692 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "038a51c29e5f47af8fe782faf7f6b6c8"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "7fe824dac7098821f9dee09f67626d67"
server_key_iv: "350a49b0535e9f3fffa39a5d47e5e333"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.407049 17474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:50.407809 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.407979 17474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:50.408085 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "038a51c29e5f47af8fe782faf7f6b6c8"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "7fe824dac7098821f9dee09f67626d67"
server_key_iv: "350a49b0535e9f3fffa39a5d47e5e333"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.408164 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:50.438359 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:50.440812 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:50.441012 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:50.441286 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:50.441617 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:50.441672 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.441746 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:50.441779 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.448133 17474 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:45421
I20260925 02:22:50.448180 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:45421 every 8 connection(s)
I20260925 02:22:50.448539 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:50.452668 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17474
I20260925 02:22:50.452769 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:50.453016 11764 external_mini_cluster.cc:1428] Setting key 55c20ef0ed23a20bd3f4cab54d48474d
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:50.458977 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.449421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:58459 (local address 127.11.125.62:35619)
0925 02:22:50.449468 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.449470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.449659 (+ 189us) server_negotiation.cc:407] Connection header received
0925 02:22:50.450613 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.450615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.450624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.450681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.451304 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.451824 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.452865 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.453191 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.453270 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.455751 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.455753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.455753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.455763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.457362 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.457918 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.457919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.457920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.457971 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.458272 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.458272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.458273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.458381 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.458432 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.458829 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.458910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:22:50.459317 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.449328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:35619 (local address 127.11.125.1:58459)
0925 02:22:50.449524 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:50.449569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:50.450521 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:50.450747 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:50.450750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:50.451014 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:50.451219 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.451223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.451910 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.451911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.452756 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.452759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.453301 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.453303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.453445 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.454032 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:50.454041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:50.455660 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:50.457451 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.457461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.457463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.457838 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.458041 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.458042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.458043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.458201 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.458453 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:50.458461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:50.458656 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 02:22:50.458802 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:22:50.459007 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260925 02:22:50.459936 17603 heartbeater.cc:347] Connected to a master server at 127.11.125.62:35619
I20260925 02:22:50.460053 17603 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:50.460263 17603 heartbeater.cc:511] Master 127.11.125.62:35619 requested a full tablet report, sending...
I20260925 02:22:50.460860 17412 ts_manager.cc:194] Registered new tserver with Master: 038a51c29e5f47af8fe782faf7f6b6c8 (127.11.125.1:45421)
I20260925 02:22:50.461539 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:58459
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:22:50.480377 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:50.562983 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:50.563184 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:50.563207 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:50.564359 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:50.564404 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:50.564430 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:50.565960 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:50.566205 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:50.566444 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:50.571512 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.571681 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.571517 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:50.571812 17608 server_base.cc:1034] running on GCE node
I20260925 02:22:50.572013 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:50.572201 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:50.573346 17608 hybrid_clock.cc:648] HybridClock initialized: now 1790302970573331 us; error 30 us; skew 500 ppm
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.576001 17608 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:22:50.576516 17608 webserver.cc:460] Webserver started at http://127.11.125.2:40767/ using document root <none> and password file <none>
I20260925 02:22:50.576812 17608 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:50.576884 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:50.577065 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:50.578104 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "d379a5a2e83d4791acb59b2e2e090e60"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "339f3733c5b39e23d1ad32c2329181df"
server_key_iv: "628ba704e79ca1083cc17da8273304ca"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.578470 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "d379a5a2e83d4791acb59b2e2e090e60"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "339f3733c5b39e23d1ad32c2329181df"
server_key_iv: "628ba704e79ca1083cc17da8273304ca"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.579905 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:50.580714 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.580948 17608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:50.581060 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "d379a5a2e83d4791acb59b2e2e090e60"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "339f3733c5b39e23d1ad32c2329181df"
server_key_iv: "628ba704e79ca1083cc17da8273304ca"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.581151 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:50.588749 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:50.591125 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:50.591306 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:50.591548 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:50.591882 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:50.591938 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.591981 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:50.592010 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.598618 17608 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:39019
I20260925 02:22:50.598667 17738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:39019 every 8 connection(s)
I20260925 02:22:50.599004 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:50.605504 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17608
I20260925 02:22:50.605635 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:50.605849 11764 external_mini_cluster.cc:1428] Setting key 19b51d19ef99b409fb8718e818bbabf5
I20260925 02:22:50.609539 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.599823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:55835 (local address 127.11.125.62:35619)
0925 02:22:50.599897 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.599898 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.600102 (+ 204us) server_negotiation.cc:407] Connection header received
0925 02:22:50.601083 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.601085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.601095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.601174 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.601783 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.602292 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.603329 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.603507 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.603559 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.606001 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.606004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.606004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.606012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.607661 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.608230 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.608232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.608233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.608315 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.608686 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.608687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.608688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.608855 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.608969 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.609300 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.609399 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:22:50.609552 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.599798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:35619 (local address 127.11.125.2:55835)
0925 02:22:50.599968 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:50.600012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:50.600982 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:50.601213 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:50.601216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:50.601490 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:50.601700 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.601704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.602384 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.602385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.603250 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.603252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.603670 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.603671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.603794 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.604272 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:50.604281 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:50.605890 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:50.607766 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.607773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.607776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.608121 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.608408 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.608409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.608409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.608529 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.608998 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:50.609012 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:50.609147 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 02:22:50.609244 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:22:50.609378 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260925 02:22:50.610117 17739 heartbeater.cc:347] Connected to a master server at 127.11.125.62:35619
I20260925 02:22:50.610230 17739 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:50.610435 17739 heartbeater.cc:511] Master 127.11.125.62:35619 requested a full tablet report, sending...
I20260925 02:22:50.610883 17412 ts_manager.cc:194] Registered new tserver with Master: d379a5a2e83d4791acb59b2e2e090e60 (127.11.125.2:39019)
I20260925 02:22:50.611317 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:55835
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:22:50.632658 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:50.716372 17744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:50.716614 17744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:50.716691 17744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:50.717888 17744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:50.717954 17744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:50.718000 17744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:50.719509 17744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:50.719794 17744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:50.720024 17744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:50.725157 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:50.725184 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:50.725234 17744 server_base.cc:1034] running on GCE node
W20260925 02:22:50.725186 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:50.725693 17744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:50.725911 17744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:50.727083 17744 hybrid_clock.cc:648] HybridClock initialized: now 1790302970727071 us; error 32 us; skew 500 ppm
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.729813 17744 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:22:50.730201 17744 webserver.cc:460] Webserver started at http://127.11.125.3:34021/ using document root <none> and password file <none>
I20260925 02:22:50.730449 17744 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:50.730495 17744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:50.730625 17744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:50.731700 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "2823b57f2f0440da8986f7d662dfd3ee"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "42569108394397950301f1bd5681114f"
server_key_iv: "d26b77363204f343d6b3e67943d9f419"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.732065 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "2823b57f2f0440da8986f7d662dfd3ee"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "42569108394397950301f1bd5681114f"
server_key_iv: "d26b77363204f343d6b3e67943d9f419"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.733474 17744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:50.734228 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.734421 17744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:50.734531 17744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "2823b57f2f0440da8986f7d662dfd3ee"
format_stamp: "Formatted at 2026-09-25 02:22:50 on dist-test-slave-hj7s"
server_key: "42569108394397950301f1bd5681114f"
server_key_iv: "d26b77363204f343d6b3e67943d9f419"
server_key_version: "encryptionkey@0"
I20260925 02:22:50.734607 17744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:50.745532 17744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:50.747938 17744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:50.748099 17744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:50.748337 17744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:50.748656 17744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:50.748710 17744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.748751 17744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:50.748780 17744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:50.755491 17744 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:34511
I20260925 02:22:50.755558 17874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:34511 every 8 connection(s)
I20260925 02:22:50.755939 17744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:50.757393 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17744
I20260925 02:22:50.757498 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:50.757701 11764 external_mini_cluster.cc:1428] Setting key 687cbb221369bdbf292bdb977cab3b65
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:22:50.766258 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.756747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:48361 (local address 127.11.125.62:35619)
0925 02:22:50.756828 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.756831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.757063 (+ 232us) server_negotiation.cc:407] Connection header received
0925 02:22:50.758439 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.758441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.758455 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.758513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.759236 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.759818 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.760793 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.761070 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.761129 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.763402 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.763403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.763403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.763410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.764866 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.765351 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.765352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.765352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.765409 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.765636 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.765637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.765637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.765747 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.765802 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.766096 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.766183 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:22:50.766326 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.756705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:35619 (local address 127.11.125.3:48361)
0925 02:22:50.756897 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:50.756987 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:50.758306 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:50.758594 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:50.758599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:50.758951 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:50.759150 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.759155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.759908 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.759908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.760711 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:50.760713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.761161 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.761162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:50.761336 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.761921 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:50.761930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:50.763294 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:50.765046 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.765053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.765054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.765282 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.765465 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:50.765465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:50.765465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:50.765562 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:50.765809 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:50.765813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:50.765932 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 02:22:50.766016 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:22:50.766140 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260925 02:22:50.766997 17875 heartbeater.cc:347] Connected to a master server at 127.11.125.62:35619
I20260925 02:22:50.767088 17875 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:50.767283 17875 heartbeater.cc:511] Master 127.11.125.62:35619 requested a full tablet report, sending...
I20260925 02:22:50.767738 17412 ts_manager.cc:194] Registered new tserver with Master: 2823b57f2f0440da8986f7d662dfd3ee (127.11.125.3:34511)
I20260925 02:22:50.768189 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:48361
I20260925 02:22:50.770007 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/test-admin.keytab.
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.785465 11764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:22:50.787149 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:50.786378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.11.125.1:45421)
0925 02:22:50.786549 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.786554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.786563 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:50.786587 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.786588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.786602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.786686 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.787059 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45014: BlockingRecv error: recv got EOF from 127.0.0.1:45014 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
W20260925 02:22:50.787288 11764 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.788280 11764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.789196 11764 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:50.797387 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.789737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.11.125.1:45421)
0925 02:22:50.789788 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.789790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.789806 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:22:50.789864 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.789865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.789874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.789925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.790292 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.791047 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.791806 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.792187 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.792262 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.794544 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.794555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.794555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.794563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.796119 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.796470 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.796473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.796474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.796600 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.796786 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.796795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.796795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.797027 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.797191 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.797233 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.797305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 02:22:50.799562 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:50.798813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.11.125.1:45421)
0925 02:22:50.798875 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.798877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.798941 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:22:50.799030 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.799031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.799047 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.799142 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.799477 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45032: BlockingRecv error: recv got EOF from 127.0.0.1:45032 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:22:50.799650 11764 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.800669 11764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.801609 11764 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:50.808650 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.802167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.11.125.1:45421)
0925 02:22:50.802227 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.802233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.802259 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:22:50.802316 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.802317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.802326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.802382 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.802755 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.803295 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.803962 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.804193 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.804263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.806347 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.806348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.806349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.806354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.807708 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.808007 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.808008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.808008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.808058 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.808209 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.808210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.808210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.808293 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.808359 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.808535 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.808590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:22:50.810354 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:50.809749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.11.125.1:45421)
0925 02:22:50.809810 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.809812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.809904 (+ 92us) server_negotiation.cc:407] Connection header received
0925 02:22:50.809973 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.809973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.809981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.810026 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.810288 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45058: BlockingRecv error: recv got EOF from 127.0.0.1:45058 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:22:50.810525 11764 init.cc:244] Error while opening credential cache '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790302942477976-11764-0/krb5kdc/krb5cc)
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.811565 11764 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:50.812497 11764 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:22:50.819363 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:50.813131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.11.125.1:45421)
0925 02:22:50.813194 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:50.813196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:50.813279 (+ 83us) server_negotiation.cc:407] Connection header received
0925 02:22:50.813318 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:50.813319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:50.813328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:50.813382 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:50.813706 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.814197 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.814875 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:50.815059 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:50.815126 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:50.817114 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:50.817115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:50.817115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:50.817121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:50.818315 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.818628 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.818629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.818629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.818683 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:50.818875 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:50.818875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:50.818876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:50.818964 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:50.819034 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:50.819120 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 02:22:50.819247 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:22:50.820036 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17474
I20260925 02:22:50.825937 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17608
I20260925 02:22:50.831002 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17744
I20260925 02:22:50.838586 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17379
2026-09-25T02:22:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17895](info): set up 2 sockets
Sep 25 02:22:50 dist-test-slave-hj7s krb5kdc[17895](info): commencing operation
krb5kdc: starting...
W20260925 02:22:51.895414 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:51 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab.
Entry for principal oryx/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab.
I20260925 02:22:52.017634 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44185
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab
--principal=oryx/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:52.099947 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:52.100164 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:52.100186 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:52.101357 17908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:52.101400 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:52.101415 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:52.101428 17908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:52.101440 17908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:52.102892 17908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab
--principal=oryx/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:52.103088 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:52.103259 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.108331 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.108278 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.108281 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.108793 17908 server_base.cc:1034] running on GCE node
I20260925 02:22:52.109073 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.109349 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.110499 17908 hybrid_clock.cc:648] HybridClock initialized: now 1790302972110455 us; error 57 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.113046 17908 init.cc:377] Logged in from keytab as oryx/127.11.125.62@KRBTEST.COM (short username oryx)
I20260925 02:22:52.113525 17908 webserver.cc:460] Webserver started at http://127.11.125.62:43031/ using document root <none> and password file <none>
I20260925 02:22:52.113758 17908 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.113831 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.113972 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:52.115059 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "13c42f340228483e95825a21b8b6c815"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "207b064fb3bdc2ced7841608801be315"
server_key_iv: "0f395aefa2d12d4d0582c6fda4497783"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.115445 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "13c42f340228483e95825a21b8b6c815"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "207b064fb3bdc2ced7841608801be315"
server_key_iv: "0f395aefa2d12d4d0582c6fda4497783"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.116816 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:52.117780 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.117962 17908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:52.118076 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "13c42f340228483e95825a21b8b6c815"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "207b064fb3bdc2ced7841608801be315"
server_key_iv: "0f395aefa2d12d4d0582c6fda4497783"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.118178 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.148653 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.150823 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:52.151021 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.155570 17908 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44185
I20260925 02:22:52.155604 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44185 every 8 connection(s)
I20260925 02:22:52.156036 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:52.157191 17976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:52.159538 17976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Bootstrap starting.
I20260925 02:22:52.160079 17976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:52.160315 17976 log.cc:824] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:52.161157 17976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: No bootstrap required, opened a new log
I20260925 02:22:52.162462 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 17908
I20260925 02:22:52.162580 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:52.162786 11764 external_mini_cluster.cc:1428] Setting key 0a512c659997e8e4fdae3c22aa31c93f
I20260925 02:22:52.163177 17976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } }
I20260925 02:22:52.163363 17976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:52.163406 17976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c42f340228483e95825a21b8b6c815, State: Initialized, Role: FOLLOWER
I20260925 02:22:52.163539 17976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } }
I20260925 02:22:52.163645 17976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:52.163679 17976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:52.163728 17976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:52.164472 17976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } }
I20260925 02:22:52.164616 17976 leader_election.cc:302] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c42f340228483e95825a21b8b6c815; no voters:
I20260925 02:22:52.164789 17976 leader_election.cc:288] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:22:52.164871 17981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:52.165043 17981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 LEADER]: Becoming Leader. State: Replica: 13c42f340228483e95825a21b8b6c815, State: Running, Role: LEADER
I20260925 02:22:52.165055 17976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:52.165165 17981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } }
I20260925 02:22:52.166157 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c42f340228483e95825a21b8b6c815. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } }
I20260925 02:22:52.166275 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:52.166380 17983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } }
I20260925 02:22:52.166456 17983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:52.167409 17997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:52.167469 17997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:52.167516 17994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:52.167663 17994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:52.170480 17994 catalog_manager.cc:1347] Generated new cluster ID: 1983e339a3484d1aaffef1c0dab67afb
I20260925 02:22:52.170536 17994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:52.173746 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.163341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.11.125.62:44185)
0925 02:22:52.163595 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.163605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.163638 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:22:52.163767 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.163774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.163832 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.163974 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.164567 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.165975 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.167048 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.167581 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.167727 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.170429 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.170433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.170433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.170443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.172190 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.172572 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.172575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.172576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.172678 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.172889 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.172890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.172890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.173174 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.173347 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.173444 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.173586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260925 02:22:52.181694 17994 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:52.182363 17994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:52.186271 17994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Generated new TSK 0
I20260925 02:22:52.186448 17994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
I20260925 02:22:52.217777 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:52.323060 18003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:52.323251 18003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:52.323278 18003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:52.324451 18003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:52.324493 18003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:52.324520 18003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:52.326087 18003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:52.326340 18003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:52.326609 18003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.332757 18009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.332749 18008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.333125 18003 server_base.cc:1034] running on GCE node
W20260925 02:22:52.332757 18011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.333396 18003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.333652 18003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.334823 18003 hybrid_clock.cc:648] HybridClock initialized: now 1790302972334807 us; error 34 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.337599 18003 init.cc:377] Logged in from keytab as oryx/127.11.125.1@KRBTEST.COM (short username oryx)
I20260925 02:22:52.338099 18003 webserver.cc:460] Webserver started at http://127.11.125.1:35763/ using document root <none> and password file <none>
I20260925 02:22:52.338351 18003 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.338423 18003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.338573 18003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:52.339775 18003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "20d93e4ae60d4eb894e381f753b6f0a3"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "5682a76a898007b4980434c79712236e"
server_key_iv: "308574d27ff676c08feb048d8f699c1b"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.340180 18003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "20d93e4ae60d4eb894e381f753b6f0a3"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "5682a76a898007b4980434c79712236e"
server_key_iv: "308574d27ff676c08feb048d8f699c1b"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.341728 18003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:52.342540 18018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.342715 18003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:52.342835 18003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "20d93e4ae60d4eb894e381f753b6f0a3"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "5682a76a898007b4980434c79712236e"
server_key_iv: "308574d27ff676c08feb048d8f699c1b"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.342912 18003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.349825 18003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.352100 18003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:52.352262 18003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.352489 18003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:52.352799 18003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:52.352854 18003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.352895 18003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:52.352960 18003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.359005 18003 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:33431
I20260925 02:22:52.359046 18133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:33431 every 8 connection(s)
I20260925 02:22:52.359408 18003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:52.363619 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18003
I20260925 02:22:52.363721 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:52.363930 11764 external_mini_cluster.cc:1428] Setting key 7ca88d40a3aa2d9eb22e1eedbd380944
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:52.369938 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.360223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:51625 (local address 127.11.125.62:44185)
0925 02:22:52.360278 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.360280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.360500 (+ 220us) server_negotiation.cc:407] Connection header received
0925 02:22:52.361099 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.361100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.361116 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.361168 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.362142 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.362666 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.363847 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.364111 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.364199 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.366926 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.366927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.366927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.366934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.368353 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.368878 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.368879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.368879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.368980 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.369271 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.369271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.369271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.369389 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.369442 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.369800 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.369873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:22:52.370268 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.360176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:51625)
0925 02:22:52.360369 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.360414 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.360991 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.361246 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.361251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.361850 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.362041 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.362046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.362772 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.362774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.363709 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.363712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.364189 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.364190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.364372 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.364963 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.364974 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.366819 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.368441 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.368449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.368451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.368803 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.369053 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.369054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.369055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.369201 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.369459 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.369467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.369645 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.369788 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.369981 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260925 02:22:52.370860 18134 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:52.370961 18134 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:52.371162 18134 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:52.371729 17940 ts_manager.cc:194] Registered new tserver with Master: 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431)
I20260925 02:22:52.372304 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:51625
WARNING: no policy specified for oryx/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
I20260925 02:22:52.391625 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:52.473618 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:52.473846 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:52.473881 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:52.475008 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:52.475073 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:52.475116 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:52.476573 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:52.476842 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:52.477128 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.482313 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.482299 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.482364 18139 server_base.cc:1034] running on GCE node
W20260925 02:22:52.482264 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.482684 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.482954 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.484117 18139 hybrid_clock.cc:648] HybridClock initialized: now 1790302972484113 us; error 36 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.486656 18139 init.cc:377] Logged in from keytab as oryx/127.11.125.2@KRBTEST.COM (short username oryx)
I20260925 02:22:52.487092 18139 webserver.cc:460] Webserver started at http://127.11.125.2:45655/ using document root <none> and password file <none>
I20260925 02:22:52.487339 18139 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.487407 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.487556 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:52.488611 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "1b3477e3986f4e72a4650574cdffef86"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "887af29fad7ce2df081dd36d92c4b25d"
server_key_iv: "7a61991aa1cc1c8a8c7fe8dc1f70da98"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.489037 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "1b3477e3986f4e72a4650574cdffef86"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "887af29fad7ce2df081dd36d92c4b25d"
server_key_iv: "7a61991aa1cc1c8a8c7fe8dc1f70da98"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.490370 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:52.491096 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.491266 18139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:52.491370 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1b3477e3986f4e72a4650574cdffef86"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "887af29fad7ce2df081dd36d92c4b25d"
server_key_iv: "7a61991aa1cc1c8a8c7fe8dc1f70da98"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.491453 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.509938 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.512418 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:52.512601 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.512869 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:52.513253 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:52.513309 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.513347 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:52.513374 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.519752 18139 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36579
I20260925 02:22:52.519812 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36579 every 8 connection(s)
I20260925 02:22:52.520171 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:52.526803 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18139
I20260925 02:22:52.526912 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:52.527105 11764 external_mini_cluster.cc:1428] Setting key a250d8b58756c8f52237f947b8ee9877
I20260925 02:22:52.530361 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.520986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:39293 (local address 127.11.125.62:44185)
0925 02:22:52.521057 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.521059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.521271 (+ 212us) server_negotiation.cc:407] Connection header received
0925 02:22:52.521836 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.521837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.521852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.521904 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.522905 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.523423 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.524527 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.524748 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.524829 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.527151 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.527153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.527154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.527162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.528755 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.529332 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.529334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.529334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.529405 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.529662 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.529663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.529664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.529813 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.529880 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.530145 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.530236 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:22:52.530489 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.520995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:39293)
0925 02:22:52.521147 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.521185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.521718 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.521995 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.522003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.522588 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.522797 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.522802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.523534 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.523535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.524424 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.524427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.524837 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.524838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.525033 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.525599 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.525608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.527054 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.528840 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.528850 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.528852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.529232 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.529486 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.529487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.529487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.529587 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.529875 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.529880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.529994 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.530071 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.530189 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260925 02:22:52.530968 18268 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:52.531070 18268 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:52.531298 18268 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:52.531760 17940 ts_manager.cc:194] Registered new tserver with Master: 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:52.532155 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:39293
WARNING: no policy specified for oryx/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
I20260925 02:22:52.551970 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:52.634907 18274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:52.635195 18274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:52.635241 18274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:52.636538 18274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:52.636612 18274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:52.636660 18274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:52.638404 18274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:52.638720 18274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:52.638993 18274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.644577 18279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.644618 18282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.644678 18274 server_base.cc:1034] running on GCE node
W20260925 02:22:52.644591 18280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.645059 18274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.645388 18274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.646543 18274 hybrid_clock.cc:648] HybridClock initialized: now 1790302972646502 us; error 55 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.649190 18274 init.cc:377] Logged in from keytab as oryx/127.11.125.3@KRBTEST.COM (short username oryx)
I20260925 02:22:52.649703 18274 webserver.cc:460] Webserver started at http://127.11.125.3:40289/ using document root <none> and password file <none>
I20260925 02:22:52.649945 18274 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.650017 18274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.650192 18274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:52.651401 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "45d419d3ab4c460d8fefc7763a297006"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "607fb85b259670e5a6577422908f582d"
server_key_iv: "7200264409bdc8eeace5ed26cc673bf5"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.651809 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "45d419d3ab4c460d8fefc7763a297006"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "607fb85b259670e5a6577422908f582d"
server_key_iv: "7200264409bdc8eeace5ed26cc673bf5"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.653399 18274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 02:22:52.654332 18289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.654554 18274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:52.654726 18274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "45d419d3ab4c460d8fefc7763a297006"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "607fb85b259670e5a6577422908f582d"
server_key_iv: "7200264409bdc8eeace5ed26cc673bf5"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.654808 18274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.677613 18274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.679896 18274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:52.680080 18274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.680344 18274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:52.680672 18274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:52.680730 18274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.680783 18274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:52.680812 18274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.687125 18274 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:37381
I20260925 02:22:52.687170 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:37381 every 8 connection(s)
I20260925 02:22:52.687525 18274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:52.696702 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.688220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:53519 (local address 127.11.125.62:44185)
0925 02:22:52.688265 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.688267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.688494 (+ 227us) server_negotiation.cc:407] Connection header received
0925 02:22:52.689035 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.689036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.689045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.689104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.690065 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.690555 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.691516 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.691690 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.691739 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.694006 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.694008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.694008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.694014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.695471 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.695908 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.695909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.695910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.695961 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.696178 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.696179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.696179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.696286 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.696336 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.696568 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.696628 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:22:52.696954 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.688182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:53519)
0925 02:22:52.688360 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.688402 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.688882 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.689163 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.689167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.689740 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.689967 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.689972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.690641 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.690642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.691442 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.691444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.691759 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.691760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.691857 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.692400 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.692409 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.693912 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.695544 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.695551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.695553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.695839 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.696023 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.696023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.696024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.696124 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.696343 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.696348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.696459 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.696540 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.696657 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260925 02:22:52.697103 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18274
I20260925 02:22:52.697187 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:52.697391 11764 external_mini_cluster.cc:1428] Setting key 4a5592710fbc5acf8c7d5e08baa57207
I20260925 02:22:52.697620 18403 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:52.697726 18403 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:52.697921 18403 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:52.698364 17940 ts_manager.cc:194] Registered new tserver with Master: 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:52.698740 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:53519
I20260925 02:22:52.709476 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.125.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.61.keytab.
Entry for principal oryx/127.11.125.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.61.keytab.
I20260925 02:22:52.733927 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:44137
--webserver_interface=127.11.125.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.61.keytab
--principal=oryx/127.11.125.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44185,127.11.125.61:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:52.817267 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:52.817488 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:52.817512 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:52.818828 18408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:52.818874 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:52.818890 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:52.818905 18408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:52.818918 18408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:52.820469 18408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44185,127.11.125.61:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.61.keytab
--principal=oryx/127.11.125.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.125.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
W20260925 02:22:52.820582 18408 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.125.62:44185,127.11.125.61:44137'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:22:52.820746 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:52.820986 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.826243 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.826270 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.826359 18408 server_base.cc:1034] running on GCE node
W20260925 02:22:52.826257 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.826658 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.826969 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.828116 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790302972828094 us; error 34 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.830745 18408 init.cc:377] Logged in from keytab as oryx/127.11.125.61@KRBTEST.COM (short username oryx)
I20260925 02:22:52.831243 18408 webserver.cc:460] Webserver started at http://127.11.125.61:44989/ using document root <none> and password file <none>
I20260925 02:22:52.831511 18408 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.831583 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.831735 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:52.832849 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/instance:
uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "ff020c63d362daeb9eef06bbe7adc53c"
server_key_iv: "c464e2b4339bf572d2d65944cafc16c7"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.833289 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal/instance:
uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "ff020c63d362daeb9eef06bbe7adc53c"
server_key_iv: "c464e2b4339bf572d2d65944cafc16c7"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.834651 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:52.835400 18423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.835596 18408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:52.835707 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "ff020c63d362daeb9eef06bbe7adc53c"
server_key_iv: "c464e2b4339bf572d2d65944cafc16c7"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.835783 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.847486 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.850198 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:52.850368 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.854593 18408 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.61:44137
I20260925 02:22:52.854640 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.61:44137 every 8 connection(s)
I20260925 02:22:52.855005 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
I20260925 02:22:52.855989 18476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:52.857641 18476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:22:52.859054 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18408
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.61@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:52.868163 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.858442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.11.125.62:44185)
0925 02:22:52.858500 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.858503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.858764 (+ 261us) server_negotiation.cc:407] Connection header received
0925 02:22:52.859491 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.859493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.859508 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.859575 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.860404 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.861174 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.862123 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.862335 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.862422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.864716 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.864718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.864718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.864724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.866582 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.867037 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.867038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.867038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.867088 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.867368 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.867369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.867369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.867578 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.867651 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.867995 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.868058 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:22:52.868353 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.858424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60284)
0925 02:22:52.858637 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.858674 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.859377 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.859645 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.859649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.860053 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.860317 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.860322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.861311 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.861312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.862014 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.862016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.862434 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.862436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.862557 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.863090 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.863100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.864632 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.866658 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.866663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.866665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.866955 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.867150 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.867151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.867151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.867288 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.867654 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.867658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.867832 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.867964 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.868123 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":22,"thread_start_us":123,"threads_started":1}
I20260925 02:22:52.868711 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.859722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.11.125.61:44137)
0925 02:22:52.859909 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.859914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.859934 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:22:52.859955 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.859957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.859978 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.860083 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.860837 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.861681 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.862415 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.862700 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.862780 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.865371 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.865380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.865380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.865387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.867217 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.867645 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.867656 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.867656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.867717 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.867902 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.867903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.867903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.868058 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.868194 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.868565 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.868644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260925 02:22:52.869015 18476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.61@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:52.874949 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.869311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39550)
0925 02:22:52.869413 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.869424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.869474 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.869582 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.869583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.869712 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.869737 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.869738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.870366 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.870367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.870831 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.870832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.871210 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.871211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.871263 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.871592 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.871596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.872683 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.874223 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.874224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.874224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.874380 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.874524 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.874525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.874525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.874566 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.874718 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.874718 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.874749 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.874800 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.874873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:22:52.874997 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.869384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.11.125.61:44137)
0925 02:22:52.869416 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.869418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.869458 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:22:52.869517 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.869518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.869526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.869579 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.869819 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.870268 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.870910 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.871125 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.871211 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.872770 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.872771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.872771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.872777 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.874156 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.874435 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.874435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.874435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.874475 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.874606 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.874606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.874607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.874684 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.874727 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.874816 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.874912 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":16}
I20260925 02:22:52.876268 18476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Bootstrap starting.
I20260925 02:22:52.877193 18476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:52.877477 18476 log.cc:824] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:52.878152 18476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: No bootstrap required, opened a new log
I20260925 02:22:52.879558 18476 raft_consensus.cc:348] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } }
I20260925 02:22:52.879702 18476 raft_consensus.cc:374] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:52.879750 18476 raft_consensus.cc:729] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241c2dea25b48e19aa47a8ec5d09e78, State: Initialized, Role: FOLLOWER
I20260925 02:22:52.879882 18476 consensus_queue.cc:260] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } }
I20260925 02:22:52.880180 18476 sys_catalog.cc:564] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:52.880143 18482 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } }
I20260925 02:22:52.880280 18482 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:22:52.882050 18493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:52.882112 18493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:52.883773 18408 master_runner.cc:428] Detected that this master e241c2dea25b48e19aa47a8ec5d09e78 is joining an existing cluster
I20260925 02:22:52.883811 18408 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.125.61:44137
I20260925 02:22:52.891893 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.885430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.11.125.61:44137)
0925 02:22:52.885467 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.885468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.885646 (+ 178us) server_negotiation.cc:407] Connection header received
0925 02:22:52.885702 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.885703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.885712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.885773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.886083 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.886524 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.887212 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.887404 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.887479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.888732 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.888734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.888734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.888743 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.890581 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.890994 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.890995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.890995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.891057 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891283 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.891284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.891284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.891422 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.891490 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.891673 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.891802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":18}
I20260925 02:22:52.892289 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.884853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.11.125.62:44185)
0925 02:22:52.884958 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.884960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.885373 (+ 413us) server_negotiation.cc:407] Connection header received
0925 02:22:52.885429 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.885430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.885442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.885508 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.885970 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.886726 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.887817 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.888097 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.888148 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.889280 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.889282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.889282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.889291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.891085 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891496 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.891498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.891498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.891551 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891759 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.891767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.891767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.891877 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.891954 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.892127 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.892197 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:22:52.892314 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.885468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39556)
0925 02:22:52.885574 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.885583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.885658 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.885772 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.885773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.885943 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.886008 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.886009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.886603 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.886604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.887133 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.887135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.887487 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.887488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.887588 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.888225 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.888229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.888614 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.890671 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.890672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.890673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.890909 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.891137 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.891139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.891209 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.891495 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.891496 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.891545 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.891636 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.892205 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260925 02:22:52.892544 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.884778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60300)
0925 02:22:52.885295 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.885307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.885364 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.885562 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.885563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.885750 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.885858 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.885860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.886839 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.886840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.887720 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.887723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.888177 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.888179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.888267 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.888695 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.888701 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.889133 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.891163 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.891165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.891402 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.891643 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.891644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.891645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.891700 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.891947 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.891948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.891989 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.892085 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.892191 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":456,"threads_started":1}
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.62@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:52.902218 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.894896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39572)
0925 02:22:52.895011 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.895086 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.895189 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.895350 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.895354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.895593 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.895638 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.895640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.896556 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.896558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.897275 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.897278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.897733 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.897734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.897798 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.898233 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.898237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.899602 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.901213 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.901220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.901222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.901412 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.901616 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.901617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.901617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.901678 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.901938 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.901944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.901987 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.902051 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.902137 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
I20260925 02:22:52.902254 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.894961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.11.125.61:44137)
0925 02:22:52.895000 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.895002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.895190 (+ 188us) server_negotiation.cc:407] Connection header received
0925 02:22:52.895272 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.895273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.895284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.895336 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.895712 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.896418 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.897355 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.897650 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.897722 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.899683 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.899684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.899685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.899692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.901138 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.901486 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.901488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.901488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.901549 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.901740 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.901740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.901741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.901873 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.901941 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.902098 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.902174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:52.903136 17940 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.125.61:44137: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
member_type: NON_VOTER
last_known_addr {
host: "127.11.125.61"
port: 44137
}
attrs {
promote: true
}
}
dest_uuid: "13c42f340228483e95825a21b8b6c815"
cas_config_opid_index: -1
I20260925 02:22:52.903469 17940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } }
I20260925 02:22:52.904003 18503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:52.904305 18450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:52.904989 18503 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.125.61:44137
I20260925 02:22:52.905218 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:52.905311 18450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:22:52.905351 18408 master.cc:537] Master@127.11.125.61:44137 shutting down...
I20260925 02:22:52.905673 18503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:52.905938 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:52.906682 17927 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 02:22:52.907159 17927 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 -> Peer e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61:44137): Couldn't send request to peer e241c2dea25b48e19aa47a8ec5d09e78. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:22:52.908315 18408 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:52.908409 18408 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:52.908475 18408 tablet_replica.cc:330] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: stopping tablet replica
I20260925 02:22:52.910984 18408 master.cc:552] Master@127.11.125.61:44137 shutdown complete.
I20260925 02:22:52.911037 18408 master_runner.cc:305] Clearing existing system tablet
I20260925 02:22:52.911170 18408 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.911218 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.911949 18408 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260925 02:22:52.912475 18507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.912611 18408 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.912673 18408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:52.912698 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "ff020c63d362daeb9eef06bbe7adc53c"
server_key_iv: "c464e2b4339bf572d2d65944cafc16c7"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.913513 18408 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:52.915233 18408 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:22:52.915308 18408 log.cc:1196] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:22:52.915514 18408 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Deleting consensus metadata
I20260925 02:22:52.915652 18408 master_runner.cc:315] Copying system tablet from 127.11.125.62:44185
I20260925 02:22:52.916304 18408 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: tablet copy: Beginning tablet copy session from remote peer at address 127.11.125.62:44185
I20260925 02:22:52.922946 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.916669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.11.125.62:44185)
0925 02:22:52.916720 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.916722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.916804 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:22:52.916849 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.916850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.916859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.916963 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.917318 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.917835 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.918900 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.919084 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.919134 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.920225 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.920227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.920227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.920233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.921854 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.922273 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.922274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.922274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.922327 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.922522 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.922522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.922522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.922618 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.922663 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.922820 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.922882 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:22:52.923033 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.916568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60314)
0925 02:22:52.916700 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:52.916712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:52.916791 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:52.916971 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:52.916972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:52.917189 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:52.917241 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.917242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.917912 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.917914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.918813 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:52.918816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.919151 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.919152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:52.919212 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.919713 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:52.919718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:52.920126 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:52.921948 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.921949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.921950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.922197 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.922392 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:52.922393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:52.922393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:52.922462 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:52.922672 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:52.922673 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:52.922723 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:22:52.922805 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:22:52.922918 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260925 02:22:52.923399 17960 tablet_copy_service.cc:140] P 13c42f340228483e95825a21b8b6c815: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e241c2dea25b48e19aa47a8ec5d09e78 ({username='oryx', principal='oryx/127.11.125.61@KRBTEST.COM'} at 127.0.0.1:60314)
I20260925 02:22:52.923508 17960 tablet_copy_service.cc:161] P 13c42f340228483e95825a21b8b6c815: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e241c2dea25b48e19aa47a8ec5d09e78 at {username='oryx', principal='oryx/127.11.125.61@KRBTEST.COM'} at 127.0.0.1:60314: session id = e241c2dea25b48e19aa47a8ec5d09e78-00000000000000000000000000000000
I20260925 02:22:52.924360 17960 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:22:52.925108 18408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:22:52.926087 18408 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: tablet copy: Starting download of 0 data blocks...
I20260925 02:22:52.926266 18408 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: tablet copy: Starting download of 1 WAL segments...
I20260925 02:22:52.927276 18408 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:22:52.928107 17960 tablet_copy_service.cc:342] P 13c42f340228483e95825a21b8b6c815: Request end of tablet copy session e241c2dea25b48e19aa47a8ec5d09e78-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.125.61@KRBTEST.COM'} at 127.0.0.1:60314
I20260925 02:22:52.928177 17960 tablet_copy_service.cc:433] P 13c42f340228483e95825a21b8b6c815: ending tablet copy session e241c2dea25b48e19aa47a8ec5d09e78-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e241c2dea25b48e19aa47a8ec5d09e78
I20260925 02:22:52.930588 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:52.931818 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.931941 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:52.933947 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:52.934001 18408 server_base.cc:1034] running on GCE node
I20260925 02:22:52.934145 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:52.934319 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:52.935448 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790302972935440 us; error 26 us; skew 500 ppm
Sep 25 02:22:52 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:52.936781 18408 init.cc:377] Logged in from keytab as oryx/127.11.125.61@KRBTEST.COM (short username oryx)
I20260925 02:22:52.937107 18408 webserver.cc:460] Webserver started at http://127.11.125.61:36405/ using document root <none> and password file <none>
I20260925 02:22:52.937274 18408 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:52.937314 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:52.937938 18408 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:52.938493 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:52.938621 18408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:52.938687 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "ff020c63d362daeb9eef06bbe7adc53c"
server_key_iv: "c464e2b4339bf572d2d65944cafc16c7"
server_key_version: "encryptionkey@0"
I20260925 02:22:52.938750 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:52.959663 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:52.961252 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:52.964435 18408 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.61:44137
I20260925 02:22:52.964550 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.61:44137 every 8 connection(s)
I20260925 02:22:52.964831 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
I20260925 02:22:52.965677 18577 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:22:52.966118 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Bootstrap starting.
I20260925 02:22:52.968394 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:22:52.968642 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: Bootstrap complete.
I20260925 02:22:52.968855 18577 raft_consensus.cc:348] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } }
I20260925 02:22:52.968979 18577 raft_consensus.cc:729] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e241c2dea25b48e19aa47a8ec5d09e78, State: Initialized, Role: LEARNER
I20260925 02:22:52.969049 18577 consensus_queue.cc:260] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } }
I20260925 02:22:52.969225 18577 sys_catalog.cc:564] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:52.969228 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:52.969372 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:22:52.973269 18592 catalog_manager.cc:1259] Loaded cluster ID: 1983e339a3484d1aaffef1c0dab67afb
I20260925 02:22:52.973330 18592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: loading cluster ID for follower catalog manager: success
I20260925 02:22:52.974820 18592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: acquiring CA information for follower catalog manager: success
I20260925 02:22:52.975462 18592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:22:52.977476 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:52.970466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.11.125.61:44137)
0925 02:22:52.970592 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:52.970594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:52.970604 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:52.970630 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:52.970631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:52.970643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:52.970756 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:52.971642 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.972271 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.973123 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:52.973360 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:52.973455 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:52.974686 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:52.974688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:52.974688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:52.974696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:52.976243 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.976623 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.976624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.976625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.976685 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:52.976891 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:52.976892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:52.976893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:52.977032 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:52.977136 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:52.977269 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:22:52.977365 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.11.125.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.60.keytab.
Entry for principal oryx/127.11.125.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.60.keytab.
I20260925 02:22:53.003461 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:34755
--webserver_interface=127.11.125.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.60.keytab
--principal=oryx/127.11.125.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:53.088063 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:53.088275 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:53.088299 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:53.089540 18596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:53.089591 18596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:53.089607 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:53.089619 18596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:53.089668 18596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:53.091174 18596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.60.keytab
--principal=oryx/127.11.125.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.125.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:53.091467 18596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:53.091713 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:53.096762 18601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.096719 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.096843 18604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.096863 18596 server_base.cc:1034] running on GCE node
I20260925 02:22:53.097232 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:53.097502 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:53.098642 18596 hybrid_clock.cc:648] HybridClock initialized: now 1790302973098614 us; error 44 us; skew 500 ppm
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:53.107488 18596 init.cc:377] Logged in from keytab as oryx/127.11.125.60@KRBTEST.COM (short username oryx)
I20260925 02:22:53.107960 18596 webserver.cc:460] Webserver started at http://127.11.125.60:38779/ using document root <none> and password file <none>
I20260925 02:22:53.108223 18596 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.108305 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.108466 18596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:53.109737 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/instance:
uuid: "3cbeff6304a0443092aa55c7cb732f24"
format_stamp: "Formatted at 2026-09-25 02:22:53 on dist-test-slave-hj7s"
server_key: "24b472ec05a47bba32eecb76d565bf40"
server_key_iv: "0a350d830f376793764dcfeaaaf7232d"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.110164 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal/instance:
uuid: "3cbeff6304a0443092aa55c7cb732f24"
format_stamp: "Formatted at 2026-09-25 02:22:53 on dist-test-slave-hj7s"
server_key: "24b472ec05a47bba32eecb76d565bf40"
server_key_iv: "0a350d830f376793764dcfeaaaf7232d"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.111590 18596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:53.112358 18611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.112536 18596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:53.112645 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "3cbeff6304a0443092aa55c7cb732f24"
format_stamp: "Formatted at 2026-09-25 02:22:53 on dist-test-slave-hj7s"
server_key: "24b472ec05a47bba32eecb76d565bf40"
server_key_iv: "0a350d830f376793764dcfeaaaf7232d"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.112725 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.139607 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:53.142146 18596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:53.142273 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:53.147490 18596 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.60:34755
I20260925 02:22:53.147594 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.60:34755 every 8 connection(s)
I20260925 02:22:53.147969 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
I20260925 02:22:53.148967 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18596
I20260925 02:22:53.149232 18665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:53.151755 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.60@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
I20260925 02:22:53.161345 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.152607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.11.125.62:44185)
0925 02:22:53.152662 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.152664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.152821 (+ 157us) server_negotiation.cc:407] Connection header received
0925 02:22:53.152860 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.152861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.152877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.152985 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.153435 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.153920 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.154728 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.154934 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.155020 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.157898 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.157899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.157900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.157908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.159612 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.160167 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.160169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.160170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.160237 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.160555 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.160556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.160556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.160713 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.160802 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.161157 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.161245 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:22:53.161580 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.152415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60316)
0925 02:22:53.152631 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.152678 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.152764 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.153019 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.153022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.153272 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.153346 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.153347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.154034 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.154036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.154624 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.154625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.155056 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.155058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.155121 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.155702 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.155713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.157792 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.159710 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.159716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.159718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.160089 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.160333 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.160334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.160334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.160480 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.160808 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.160816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.161008 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.161102 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.161269 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260925 02:22:53.161594 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.149741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.11.125.60:34755)
0925 02:22:53.149914 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.149923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.149950 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:22:53.150676 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.150684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.150737 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.150904 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.151538 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.153072 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.153965 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.154620 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.154799 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.158204 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.158212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.158213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.158219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.160273 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.160666 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.160669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.160669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.160736 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.160963 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.160964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.160964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.161134 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.161293 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.161390 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.161480 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260925 02:22:53.162307 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.60@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:53.168428 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.162624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39600)
0925 02:22:53.162674 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.162688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.162729 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.162862 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.162863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.162971 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.163030 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.163031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.163656 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.163656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.164139 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.164140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.164488 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.164489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.164535 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.164960 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.164966 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.166201 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.167607 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.167608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.167608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.167767 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.167947 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.167948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.167948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.167997 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.168201 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.168202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.168236 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.168287 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.168363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:22:53.168592 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.162711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.11.125.61:44137)
0925 02:22:53.162755 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.162757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.162766 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:53.162789 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.162789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.162816 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.162860 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.163093 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.163590 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.164215 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.164423 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.164474 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.166286 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.166288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.166289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.166296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.167531 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.167836 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.167837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.167837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.167889 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.168048 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.168048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.168048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.168148 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.168198 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.168343 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.168449 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:53.169004 18665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.60@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
I20260925 02:22:53.175064 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.169281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39798)
0925 02:22:53.169331 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.169342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.169379 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.169483 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.169484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.169651 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.169685 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.169687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.170391 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.170391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.170898 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.170900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.171245 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.171246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.171296 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.171633 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.171637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.172714 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.174307 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.174308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.174308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.174458 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.174609 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.174610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.174610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.174657 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.174829 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.174830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.174863 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.174912 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.174984 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:22:53.175115 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.169332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.11.125.60:34755)
0925 02:22:53.169396 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.169397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.169404 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:22:53.169427 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.169427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.169435 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.169482 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.169763 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.170323 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.170968 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.171180 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.171235 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.172787 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.172788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.172789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.172795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.174215 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.174514 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.174515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.174515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.174558 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.174693 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.174693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.174694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.174778 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.174823 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.174927 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.175025 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":47}
I20260925 02:22:53.176810 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Bootstrap starting.
I20260925 02:22:53.177798 18665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:53.178393 18665 log.cc:824] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:53.179404 18665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: No bootstrap required, opened a new log
I20260925 02:22:53.181494 18665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } }
I20260925 02:22:53.181625 18665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:53.181670 18665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbeff6304a0443092aa55c7cb732f24, State: Initialized, Role: FOLLOWER
I20260925 02:22:53.181807 18665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } }
I20260925 02:22:53.182116 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } } }
I20260925 02:22:53.182168 18665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:53.182228 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:22:53.184116 18681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:22:53.184209 18681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:22:53.186473 18596 master_runner.cc:428] Detected that this master 3cbeff6304a0443092aa55c7cb732f24 is joining an existing cluster
I20260925 02:22:53.186534 18596 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.125.60:34755
I20260925 02:22:53.195612 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.188128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.11.125.60:34755)
0925 02:22:53.188164 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.188166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.188663 (+ 497us) server_negotiation.cc:407] Connection header received
0925 02:22:53.188697 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.188698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.188708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.188789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.189643 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.190212 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.190963 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.191228 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.191299 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.192549 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.192550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.192550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.192557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.194378 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.194784 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.194786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.194786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.194861 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.195055 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.195056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.195056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.195165 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.195237 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.195406 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.195504 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260925 02:22:53.195701 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.187923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39804)
0925 02:22:53.188582 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.188593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.188655 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.188865 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.188866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.189311 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.189556 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.189558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.190296 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.190297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.190884 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.190886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.191427 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.191428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.191529 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.192059 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.192064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.192446 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.194466 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.194467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.194467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.194641 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.194925 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.194926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.194926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.194985 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.195229 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.195230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.195269 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.195382 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.195560 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":80,"thread_start_us":129,"threads_started":1}
I20260925 02:22:53.200340 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.188136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.11.125.61:44137)
0925 02:22:53.188180 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.188182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.193234 (+ 5052us) server_negotiation.cc:407] Connection header received
0925 02:22:53.193353 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.193355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.193367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.193433 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.194160 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.194938 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.196396 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.196603 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.196660 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.197908 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.197909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.197910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.197917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.199226 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.199609 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.199610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.199611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.199667 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.199877 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.199878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.199878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.199979 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.200031 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.200213 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.200274 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:53.200484 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.188319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39604)
0925 02:22:53.193140 (+ 4821us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.193152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.193280 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.193451 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.193452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.193871 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.194059 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.194061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.195075 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.195076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.196308 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.196311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.196681 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.196682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.196802 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.197286 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.197291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.197816 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.199311 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.199312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.199313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.199534 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.199735 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.199736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.199736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.199816 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.200068 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.200069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.200113 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.200198 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.200309 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4777,"thread_start_us":113,"threads_started":1}
I20260925 02:22:53.212836 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.187946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.11.125.62:44185)
0925 02:22:53.188027 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.188029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.188375 (+ 346us) server_negotiation.cc:407] Connection header received
0925 02:22:53.188454 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.188455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.188466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.188517 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.189261 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.189905 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.191031 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.191353 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.191446 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.210014 (+ 18568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.210016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.210016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.210034 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.211532 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.211995 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.211997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.211997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.212062 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.212271 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.212271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.212271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.212393 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.212444 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.212621 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.212715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:22:53.212936 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.187869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60324)
0925 02:22:53.188297 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.188307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.188389 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.188518 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.188519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.189065 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.189176 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.189178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.189993 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.189994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.190944 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.190946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.191457 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.191458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.191592 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.209413 (+ 17821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.209420 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.209877 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.211651 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.211653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.211653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.211896 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.212135 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.212136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.212136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.212208 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.212457 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.212458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.212512 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.212587 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.212800 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":99,"threads_started":1}
I20260925 02:22:53.213280 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18003
I20260925 02:22:53.220357 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:33431
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:44185
--webserver_port=35763
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302972, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.62@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
I20260925 02:22:53.224357 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.217479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39806)
0925 02:22:53.217547 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.217558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.217607 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.217746 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.217747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.217915 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.217959 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.217960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.218588 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.218589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.219130 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.219132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.219516 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.219517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.219586 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.219969 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.219974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.221666 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.223335 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.223336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.223337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.223511 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.223720 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.223721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.223721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.223784 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.224033 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.224034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.224081 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.224165 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.224274 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:22:53.224747 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.217512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.11.125.60:34755)
0925 02:22:53.217561 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.217563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.217590 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:22:53.217679 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.217680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.217689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.217734 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.218036 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.218505 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.219218 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.219441 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.219520 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.221775 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.221777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.221777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.221799 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.223247 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.223584 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.223585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.223585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.223640 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.223852 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.223853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.223853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.223966 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.224039 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.224586 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.224660 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:22:53.225327 17940 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.125.60:34755: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24"
member_type: NON_VOTER
last_known_addr {
host: "127.11.125.60"
port: 34755
}
attrs {
promote: true
}
}
dest_uuid: "13c42f340228483e95825a21b8b6c815"
cas_config_opid_index: 5
I20260925 02:22:53.225606 17940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
I20260925 02:22:53.225962 18503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.226217 18638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:53.227017 18503 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.125.60:34755
I20260925 02:22:53.227141 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.227207 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.227193 18638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:22:53.227591 18503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:53.227957 18596 master.cc:537] Master@127.11.125.60:34755 shutting down...
I20260925 02:22:53.229948 18638 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
W20260925 02:22:53.230037 18638 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
I20260925 02:22:53.230616 18638 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:53.231844 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c42f340228483e95825a21b8b6c815. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:53.231949 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:22:53.231985 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:53.232016 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:22:53.232041 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } }
I20260925 02:22:53.232080 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 02:22:53.232492 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.232605 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
W20260925 02:22:53.233340 17925 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 -> Peer 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60:34755): Couldn't send request to peer 3cbeff6304a0443092aa55c7cb732f24. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:22:53.233778 18596 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 02:22:53.233860 18596 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 02:22:53.233891 18596 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: stopping tablet replica
I20260925 02:22:53.236666 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.227886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.11.125.61:44137)
0925 02:22:53.233026 (+ 5140us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.233029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.233181 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:22:53.233210 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.233211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.233238 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.233294 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.233634 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.234364 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.235559 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.236158 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.236230 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.236314 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.236546 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.236595 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5108}
I20260925 02:22:53.237329 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.227569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39610)
0925 02:22:53.233060 (+ 5491us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.233072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.233128 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.233363 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.233364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.233366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.233534 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.233538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.234441 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.234442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.235458 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.235461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.236353 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.236354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.236425 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.236451 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.236879 (+ 428us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.236970 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5475}
I20260925 02:22:53.237627 18551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 LEARNER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:22:53.237848 18502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:22:53.238415 18551 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.239729 18502 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: attempting to promote NON_VOTER e241c2dea25b48e19aa47a8ec5d09e78 to VOTER
I20260925 02:22:53.239778 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c42f340228483e95825a21b8b6c815. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.239874 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:22:53.239909 18581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: NON_VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: true } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.239951 18581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:22:53.239935 18502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
I20260925 02:22:53.240592 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.240366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39820)
0925 02:22:53.240485 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.240536 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":77,"threads_started":1}
I20260925 02:22:53.240605 18551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 LEARNER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:22:53.241027 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.240928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39834)
0925 02:22:53.240966 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.240994 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 02:22:53.241148 17925 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 -> Peer 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60:34755): Couldn't send request to peer 3cbeff6304a0443092aa55c7cb732f24. Status: Network error: Client connection negotiation failed: client connection to 127.11.125.60:34755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:22:53.241379 18700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:22:53.241955 18503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.242719 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.242769 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.242839 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.242807 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.243006 18551 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.243803 18699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.243887 18699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:22:53.252341 18596 master.cc:552] Master@127.11.125.60:34755 shutdown complete.
I20260925 02:22:53.252419 18596 master_runner.cc:305] Clearing existing system tablet
I20260925 02:22:53.252570 18596 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.252638 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.253472 18596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:22:53.254047 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.254225 18596 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.254290 18596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:53.254323 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "3cbeff6304a0443092aa55c7cb732f24"
format_stamp: "Formatted at 2026-09-25 02:22:53 on dist-test-slave-hj7s"
server_key: "24b472ec05a47bba32eecb76d565bf40"
server_key_iv: "0a350d830f376793764dcfeaaaf7232d"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.254997 18596 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:53.256397 18596 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:22:53.256484 18596 log.cc:1196] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:22:53.256735 18596 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Deleting consensus metadata
I20260925 02:22:53.256886 18596 master_runner.cc:315] Copying system tablet from 127.11.125.62:44185
I20260925 02:22:53.257601 18596 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: tablet copy: Beginning tablet copy session from remote peer at address 127.11.125.62:44185
I20260925 02:22:53.265013 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.258340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.11.125.62:44185)
0925 02:22:53.258395 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.258397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.258534 (+ 137us) server_negotiation.cc:407] Connection header received
0925 02:22:53.258607 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.258608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.258620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.258691 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.259078 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.259695 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.260479 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.260711 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.260793 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.262283 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.262285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.262285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.262294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.263744 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.264114 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.264116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.264116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.264180 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.264383 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.264384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.264385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.264509 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.264590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.264758 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.264847 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:22:53.265432 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.258307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60330)
0925 02:22:53.258446 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.258457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.258549 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.258698 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.258699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.258919 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.258986 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.258988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.259792 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.259793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.260336 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.260338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.260794 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.260795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.260923 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.261842 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.261847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.262149 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.263852 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.263854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.263854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.264040 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.264250 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.264250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.264251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.264313 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.264581 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.264582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.264631 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.265218 (+ 587us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.265341 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260925 02:22:53.265690 17960 tablet_copy_service.cc:140] P 13c42f340228483e95825a21b8b6c815: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3cbeff6304a0443092aa55c7cb732f24 ({username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:60330)
I20260925 02:22:53.265771 17960 tablet_copy_service.cc:161] P 13c42f340228483e95825a21b8b6c815: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3cbeff6304a0443092aa55c7cb732f24 at {username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:60330: session id = 3cbeff6304a0443092aa55c7cb732f24-00000000000000000000000000000000
I20260925 02:22:53.266458 17960 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:22:53.267002 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:22:53.268014 18596 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: tablet copy: Starting download of 0 data blocks...
I20260925 02:22:53.268185 18596 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: tablet copy: Starting download of 1 WAL segments...
I20260925 02:22:53.268975 18596 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:22:53.269843 17960 tablet_copy_service.cc:342] P 13c42f340228483e95825a21b8b6c815: Request end of tablet copy session 3cbeff6304a0443092aa55c7cb732f24-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:60330
I20260925 02:22:53.269903 17960 tablet_copy_service.cc:433] P 13c42f340228483e95825a21b8b6c815: ending tablet copy session 3cbeff6304a0443092aa55c7cb732f24-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3cbeff6304a0443092aa55c7cb732f24
I20260925 02:22:53.272512 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:53.273795 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.274093 18596 server_base.cc:1034] running on GCE node
W20260925 02:22:53.274276 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.277376 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.277549 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:53.277733 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:53.278857 18596 hybrid_clock.cc:648] HybridClock initialized: now 1790302973278840 us; error 38 us; skew 500 ppm
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:53.280254 18596 init.cc:377] Logged in from keytab as oryx/127.11.125.60@KRBTEST.COM (short username oryx)
I20260925 02:22:53.280558 18596 webserver.cc:460] Webserver started at http://127.11.125.60:33243/ using document root <none> and password file <none>
I20260925 02:22:53.280684 18596 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.280721 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.281502 18596 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.282083 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.282335 18596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:53.282426 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "3cbeff6304a0443092aa55c7cb732f24"
format_stamp: "Formatted at 2026-09-25 02:22:53 on dist-test-slave-hj7s"
server_key: "24b472ec05a47bba32eecb76d565bf40"
server_key_iv: "0a350d830f376793764dcfeaaaf7232d"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.282491 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.303145 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:53.305037 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:53.308231 18596 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.60:34755
I20260925 02:22:53.308568 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
I20260925 02:22:53.308975 18773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.60:34755 every 8 connection(s)
I20260925 02:22:53.309485 18774 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:22:53.309898 18774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Bootstrap starting.
I20260925 02:22:53.311823 18774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:22:53.312068 18774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Bootstrap complete.
I20260925 02:22:53.312268 18774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
I20260925 02:22:53.312371 18774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3cbeff6304a0443092aa55c7cb732f24, State: Initialized, Role: LEARNER
I20260925 02:22:53.312418 18774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } }
I20260925 02:22:53.312628 18774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:53.312601 18777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.312810 18777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:22:53.314139 18788 catalog_manager.cc:1259] Loaded cluster ID: 1983e339a3484d1aaffef1c0dab67afb
I20260925 02:22:53.314230 18788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: loading cluster ID for follower catalog manager: success
I20260925 02:22:53.315834 18788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: acquiring CA information for follower catalog manager: success
I20260925 02:22:53.316125 18788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:22:53.342912 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:53.343178 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:53.343217 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:53.344878 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:53.344959 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:53.345000 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:53.347155 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:33431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=35763
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:53.347409 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:53.347673 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:53.353176 18796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.353240 18794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.353384 18690 server_base.cc:1034] running on GCE node
W20260925 02:22:53.353178 18793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.353709 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:53.353967 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:53.355135 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790302973355110 us; error 41 us; skew 500 ppm
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:53.357831 18690 init.cc:377] Logged in from keytab as oryx/127.11.125.1@KRBTEST.COM (short username oryx)
I20260925 02:22:53.358345 18690 webserver.cc:460] Webserver started at http://127.11.125.1:35763/ using document root <none> and password file <none>
I20260925 02:22:53.358587 18690 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.358672 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.360157 18690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:22:53.360810 18803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.360992 18690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:53.361100 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "20d93e4ae60d4eb894e381f753b6f0a3"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "5682a76a898007b4980434c79712236e"
server_key_iv: "308574d27ff676c08feb048d8f699c1b"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.361483 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.382282 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:53.384306 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:53.384480 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:53.384740 18690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:53.385138 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:53.385195 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.385239 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:53.385267 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.391592 18690 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:33431
I20260925 02:22:53.391637 18917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:33431 every 8 connection(s)
I20260925 02:22:53.392063 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:53.396461 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18690
I20260925 02:22:53.396606 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18139
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
I20260925 02:22:53.406002 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.393577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:40103 (local address 127.11.125.61:44137)
0925 02:22:53.393627 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.393630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.393694 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:22:53.394818 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.394820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.394834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.394869 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.395875 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.396622 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.398644 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.398959 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.398995 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.402203 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.402205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.402205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.402213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.403974 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.404758 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.404760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.404760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.404837 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.405316 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.405317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.405318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.405465 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.405507 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.405823 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.405901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:22:53.406002 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.393247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33919 (local address 127.11.125.62:44185)
0925 02:22:53.393324 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.393328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.393344 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:22:53.394826 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.394829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.394842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.394874 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.395842 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.396655 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.398644 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.398947 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.398992 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.402629 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.402631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.402631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.402640 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.404185 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.404836 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.404837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.404837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.404896 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.405227 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.405228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.405229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.405387 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.405441 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.405811 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.405901 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:22:53.406629 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.392889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:33919)
0925 02:22:53.393193 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.393240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.394688 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.394965 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.394970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.395513 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.395721 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.395726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.397197 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.397198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.398532 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.398535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.399089 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.399089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.399236 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.399856 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.399866 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.402522 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.404288 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.404289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.404290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.404751 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.404991 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.404992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.404993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.405130 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.405476 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.405483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.405650 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.406138 (+ 488us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.406398 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260925 02:22:53.406948 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.393389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.11.125.1:40103)
0925 02:22:53.393632 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.393650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.394693 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.394965 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.394970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.395552 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.395729 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.395732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.397590 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.397591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.398532 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.398535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.399087 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.399088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.399236 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.400140 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.400146 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.402097 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.404065 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.404073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.404076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.404661 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.405172 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.405173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.405174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.405233 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.405549 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.405550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.405650 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.406481 (+ 831us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.406569 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":18,"thread_start_us":67,"threads_started":1}
I20260925 02:22:53.407122 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:36579
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:44185
--webserver_port=45655
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:22:53.407109 18918 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:53.407465 18918 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.407686 18918 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:53.407701 18920 heartbeater.cc:347] Connected to a master server at 127.11.125.61:44137
I20260925 02:22:53.407752 18920 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.407809 18920 heartbeater.cc:511] Master 127.11.125.61:44137 requested a full tablet report, sending...
I20260925 02:22:53.407876 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.392998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:56043 (local address 127.11.125.60:34755)
0925 02:22:53.393757 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.393759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.393769 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:53.394798 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.394800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.394814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.394884 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.395842 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.397612 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.398993 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.399387 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.399451 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.403199 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.403200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.403200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.403208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.405934 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.406471 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.406472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.406473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.406535 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.406742 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.406743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.406743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.406883 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.406949 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.407632 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.407739 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":55,"threads_started":1}
I20260925 02:22:53.408145 17940 ts_manager.cc:194] Re-registered known tserver with Master: 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431)
I20260925 02:22:53.408267 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.392890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.11.125.1:56043)
0925 02:22:53.393567 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.393580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.394688 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.394965 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.394969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.395561 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.395743 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.395744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.397713 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.397715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.398887 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.398891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.399512 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.399513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.399659 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.400157 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.400162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.403080 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.406026 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.406027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.406028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.406384 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.406602 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.406603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.406603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.406679 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.406979 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.406980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.407025 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.407086 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.407236 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260925 02:22:53.408682 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:33919
I20260925 02:22:53.408725 18919 heartbeater.cc:347] Connected to a master server at 127.11.125.60:34755
I20260925 02:22:53.408771 18919 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.408887 18919 heartbeater.cc:511] Master 127.11.125.60:34755 requested a full tablet report, sending...
I20260925 02:22:53.409137 18541 ts_manager.cc:194] Registered new tserver with Master: 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431)
I20260925 02:22:53.409392 18738 ts_manager.cc:194] Registered new tserver with Master: 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431)
W20260925 02:22:53.493121 18926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:53.493387 18926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:53.493412 18926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:53.494560 18926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:53.494596 18926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:53.494621 18926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:53.496014 18926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:36579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=45655
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:53.496205 18926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:53.496384 18926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:53.501418 18934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.501428 18931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.501428 18932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.501446 18926 server_base.cc:1034] running on GCE node
I20260925 02:22:53.501816 18926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:53.502008 18926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:53.503144 18926 hybrid_clock.cc:648] HybridClock initialized: now 1790302973503122 us; error 36 us; skew 500 ppm
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:53.505555 18926 init.cc:377] Logged in from keytab as oryx/127.11.125.2@KRBTEST.COM (short username oryx)
I20260925 02:22:53.506027 18926 webserver.cc:460] Webserver started at http://127.11.125.2:45655/ using document root <none> and password file <none>
I20260925 02:22:53.506222 18926 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.506273 18926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.507647 18926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:53.508206 18941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.508387 18926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.508491 18926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1b3477e3986f4e72a4650574cdffef86"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "887af29fad7ce2df081dd36d92c4b25d"
server_key_iv: "7a61991aa1cc1c8a8c7fe8dc1f70da98"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.508874 18926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.533341 18926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:53.535333 18926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:53.535513 18926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:53.535769 18926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:53.536108 18926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:53.536163 18926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.536199 18926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:53.536248 18926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.542940 18926 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36579
I20260925 02:22:53.542981 19055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36579 every 8 connection(s)
I20260925 02:22:53.543354 18926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:53.553258 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 18926
I20260925 02:22:53.553421 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18274
I20260925 02:22:53.554667 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:51979 (local address 127.11.125.60:34755)
0925 02:22:53.544440 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.544442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.544451 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:53.545588 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.545590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.545603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.545636 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.546448 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547271 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548413 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548620 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548651 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.550949 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.550950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.550951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.550958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.552852 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553473 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.553474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.553474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.553530 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553809 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.553810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.553811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.553952 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.554000 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.554480 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.554573 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:53.554817 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:54059 (local address 127.11.125.62:44185)
0925 02:22:53.544639 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.544641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.544650 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:53.545590 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.545591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.545603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.545636 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.546449 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547263 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548441 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548759 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.551389 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.551390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.551391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.551399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.552959 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553672 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.553673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.553673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.553736 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553950 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.553951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.553951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.554092 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.554138 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.554630 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.554711 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:22:53.554947 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.11.125.2:51979)
0925 02:22:53.544248 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.544298 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.545461 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.545720 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.545725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.546129 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.546335 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.546341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547366 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.547367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.548298 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.548300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548744 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.548809 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.549352 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.549361 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.550843 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.552997 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.553008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.553376 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.553600 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.553601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.553739 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.554052 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.554059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.554285 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.554441 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.554646 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":53,"thread_start_us":119,"threads_started":1}
I20260925 02:22:53.555208 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:54059)
0925 02:22:53.544571 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.544581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.545473 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.545720 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.545725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.546126 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.546334 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.546341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547366 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.547367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.548298 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.548300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548843 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.548962 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.549354 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.549362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.551273 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.553038 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.553040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.553568 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.553812 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.553812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.553813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.553875 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.554473 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.554474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.554514 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.554963 (+ 449us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.555084 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":55,"thread_start_us":55,"threads_started":1}
I20260925 02:22:53.556372 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:33675 (local address 127.11.125.61:44137)
0925 02:22:53.544121 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.544124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.544399 (+ 275us) server_negotiation.cc:407] Connection header received
0925 02:22:53.545568 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.545569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.545582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.545636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.546435 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547020 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548058 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548258 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548337 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.552293 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.552295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.552295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.552302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.554097 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.555420 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.555421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.555421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.555498 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.555752 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.555754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.555755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.555921 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.555971 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.556197 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.556281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:22:53.556469 19057 heartbeater.cc:347] Connected to a master server at 127.11.125.60:34755
I20260925 02:22:53.556619 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.544077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.11.125.2:33675)
0925 02:22:53.544255 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.544297 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.545461 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.545724 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.545725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.546116 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.546332 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.546337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.547137 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.547138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.547960 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.547962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.548346 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.548347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.548515 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.549354 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.549362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.551905 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.554167 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.554168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.554168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.554360 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.555577 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.555578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.555579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.555656 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.556021 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.556021 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.556067 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.556149 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.556278 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":30,"thread_start_us":89,"threads_started":1}
I20260925 02:22:53.556676 19057 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.556876 19057 heartbeater.cc:511] Master 127.11.125.60:34755 requested a full tablet report, sending...
I20260925 02:22:53.557368 18738 ts_manager.cc:194] Registered new tserver with Master: 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:53.558573 19056 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:53.558627 19056 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.558640 19058 heartbeater.cc:347] Connected to a master server at 127.11.125.61:44137
I20260925 02:22:53.558700 19058 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.558701 19056 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:53.558799 19058 heartbeater.cc:511] Master 127.11.125.61:44137 requested a full tablet report, sending...
I20260925 02:22:53.558969 17940 ts_manager.cc:194] Re-registered known tserver with Master: 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:53.559099 18541 ts_manager.cc:194] Registered new tserver with Master: 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:53.559423 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:54059
I20260925 02:22:53.561604 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:37381
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:44185
--webserver_port=40289
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:53.646306 19063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:53.646524 19063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:53.646564 19063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:53.647716 19063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:53.647754 19063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:53.647779 19063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:53.649379 19063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=40289
--tserver_master_addrs=127.11.125.62:44185,127.11.125.61:44137,127.11.125.60:34755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:53.649585 19063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:53.649776 19063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:53.654807 19071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.654829 19068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:53.654815 19069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:53.654997 19063 server_base.cc:1034] running on GCE node
I20260925 02:22:53.655300 19063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:53.655550 19063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:53.656747 19063 hybrid_clock.cc:648] HybridClock initialized: now 1790302973656729 us; error 39 us; skew 500 ppm
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:53.659334 19063 init.cc:377] Logged in from keytab as oryx/127.11.125.3@KRBTEST.COM (short username oryx)
I20260925 02:22:53.659828 19063 webserver.cc:460] Webserver started at http://127.11.125.3:40289/ using document root <none> and password file <none>
I20260925 02:22:53.660048 19063 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:53.660102 19063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:53.661525 19063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:53.662168 19078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.662369 19063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:53.662456 19063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "45d419d3ab4c460d8fefc7763a297006"
format_stamp: "Formatted at 2026-09-25 02:22:52 on dist-test-slave-hj7s"
server_key: "607fb85b259670e5a6577422908f582d"
server_key_iv: "7200264409bdc8eeace5ed26cc673bf5"
server_key_version: "encryptionkey@0"
I20260925 02:22:53.662801 19063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:53.674458 19063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:53.676263 19063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:53.676412 19063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:53.676652 19063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:22:53.677031 19063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:53.677068 19063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.677091 19063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:53.677134 19063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:53.683139 19063 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:37381
I20260925 02:22:53.683235 19192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:37381 every 8 connection(s)
I20260925 02:22:53.683578 19063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:53.686167 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 19063
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.60@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
Sep 25 02:22:53 dist-test-slave-hj7s krb5kdc[17895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302973, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.61@KRBTEST.COM
I20260925 02:22:53.696847 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.11.125.3:42421)
0925 02:22:53.685072 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.685082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.685791 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.686063 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.686064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.686450 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.686669 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.686677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.687781 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.687781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.688632 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.688635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689197 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.689335 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.689953 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.689963 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.692944 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.694982 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.694990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.694993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.695328 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.695552 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.695553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.695554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.695697 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.696031 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.696039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.696207 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.696338 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.696529 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260925 02:22:53.697140 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:47399 (local address 127.11.125.62:44185)
0925 02:22:53.684345 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.684347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.684634 (+ 287us) server_negotiation.cc:407] Connection header received
0925 02:22:53.685876 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.685877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.685892 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.685958 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.686770 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.687561 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689539 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689847 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689911 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.692560 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.692561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.692561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.692569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.695668 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.696097 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.696098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.696098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.696160 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.696358 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.696359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.696359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.696491 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.696548 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.696956 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.697035 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:22:53.697211 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:47399)
0925 02:22:53.684492 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.684535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.685762 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.686055 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.686059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.686447 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.686669 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.686673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.687698 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.687699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.689471 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.689473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689967 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.690054 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.690457 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.690462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.692447 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.695765 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.695766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.695766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.696007 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.696229 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.696230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.696230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.696286 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.696576 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.696577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.696626 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.696714 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.696838 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":808,"thread_start_us":97,"threads_started":1}
I20260925 02:22:53.697944 19193 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44185
I20260925 02:22:53.698050 19193 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.698248 19193 heartbeater.cc:511] Master 127.11.125.62:44185 requested a full tablet report, sending...
I20260925 02:22:53.698652 17940 ts_manager.cc:194] Re-registered known tserver with Master: 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:53.698843 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:42421 (local address 127.11.125.60:34755)
0925 02:22:53.684360 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.684362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.685160 (+ 798us) server_negotiation.cc:407] Connection header received
0925 02:22:53.685946 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.685948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.685959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.686009 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.686808 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.687562 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.688734 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689067 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689131 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.693037 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.693038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.693038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.693046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.694876 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.695418 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.695419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.695419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.695482 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.695780 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.695781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.695781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.695933 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.695999 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.698668 (+ 2669us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.698750 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:53.699142 17940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:47399
I20260925 02:22:53.699342 19194 heartbeater.cc:347] Connected to a master server at 127.11.125.60:34755
I20260925 02:22:53.699352 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.687216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.11.125.62:44185)
0925 02:22:53.689591 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.689593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.689599 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:22:53.689615 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.689616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.689627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.689686 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.690762 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.691521 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.693339 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.694039 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.694115 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.695456 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.695457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.695458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.695465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.697850 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.698243 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.698244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.698245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.698305 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.698523 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.698524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.698524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.698657 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.698750 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.699153 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.699239 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2349,"thread_start_us":100,"threads_started":1}
I20260925 02:22:53.699397 19194 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.699465 19194 heartbeater.cc:511] Master 127.11.125.60:34755 requested a full tablet report, sending...
I20260925 02:22:53.699887 18738 ts_manager.cc:194] Registered new tserver with Master: 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:53.701363 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:41975 (local address 127.11.125.61:44137)
0925 02:22:53.684771 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.684773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.685247 (+ 474us) server_negotiation.cc:407] Connection header received
0925 02:22:53.685892 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.685894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.685907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.685942 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.687653 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.688177 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.688842 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689141 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689218 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.693037 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.693038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.693039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.693047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.698025 (+ 4978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.700342 (+ 2317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.700343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.700343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.700405 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.700766 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.700767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.700767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.700933 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.700996 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.701163 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.701265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:22:53.701392 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.684592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.11.125.3:41975)
0925 02:22:53.685185 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.685192 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.685784 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.687138 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.687139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.687494 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:53.687550 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.687552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.688255 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.688256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.688755 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.688757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.689217 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.689218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.689334 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.690839 (+ 1505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:53.690844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:53.692944 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:53.698321 (+ 5377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.698322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.698322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.698565 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.700476 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:53.700477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:53.700477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:53.700536 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:53.701004 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:53.701005 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:53.701052 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.701130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.701254 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":58,"threads_started":1}
I20260925 02:22:53.701704 19195 heartbeater.cc:347] Connected to a master server at 127.11.125.61:44137
I20260925 02:22:53.701753 19195 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:53.701844 19195 heartbeater.cc:511] Master 127.11.125.61:44137 requested a full tablet report, sending...
I20260925 02:22:53.702157 18541 ts_manager.cc:194] Registered new tserver with Master: 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:53.702505 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.690715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.11.125.60:34755)
0925 02:22:53.691422 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.691424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.691780 (+ 356us) server_negotiation.cc:407] Connection header received
0925 02:22:53.691805 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.691806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.691820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.691877 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.692586 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.693383 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.694447 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.694782 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.694849 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.696441 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.696442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.696442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.696449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.700704 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.701123 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.701124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.701125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.701196 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.701575 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.701576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.701576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.701737 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.701860 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.702309 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.702403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":64,"threads_started":1}
I20260925 02:22:53.701462 17940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:53.703667 17940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:53.705893 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.687677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.11.125.61:44137)
0925 02:22:53.687797 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.687799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.690351 (+ 2552us) server_negotiation.cc:407] Connection header received
0925 02:22:53.691034 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.691035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.691049 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.691091 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.691936 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.698831 (+ 6895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.700280 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.700592 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.700657 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.703085 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.703086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.703087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.703094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.704693 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.705086 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.705086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.705087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.705136 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.705328 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.705329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.705329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.705473 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.705567 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.705677 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.705755 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260925 02:22:53.712003 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:33431 (local address 127.0.0.1:42950)
0925 02:22:53.708405 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.708416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.708456 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.708987 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.708988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.708990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.709034 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.709036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.709956 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.709957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.710788 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.710790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.711784 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.711785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.711840 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.711843 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.711885 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.711931 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 02:22:53.712291 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.11.125.1:33431)
0925 02:22:53.708745 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.708750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.708769 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:22:53.708819 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.708822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.708841 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.708991 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.709094 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.709865 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.710888 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.711686 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.711782 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.711922 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.711953 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.711996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260925 02:22:53.712786 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:37381 (local address 127.0.0.1:56428)
0925 02:22:53.708523 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.708532 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.708595 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.709180 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.709180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.709182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.709214 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.709215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.710465 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.710467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.711459 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.711462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.712573 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.712574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.712631 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.712636 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.712667 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.712714 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260925 02:22:53.712872 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36579 (local address 127.0.0.1:44030)
0925 02:22:53.708216 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.708226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.708279 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.709213 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.709214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.709217 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.709272 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.709274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.710558 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.710559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.711652 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.711656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.712601 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.712603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.712666 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.712671 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.712720 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.712757 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:22:53.713094 18851 tablet_service.cc:1467] Processing CreateTablet for tablet e97317237fd34a08b77fb2d18f142f79 (DEFAULT_TABLE table=test-table [id=2470e14a7ff64c53afdfeebcbff12e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:53.713270 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.11.125.3:37381)
0925 02:22:53.709029 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.709032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.709041 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:53.709082 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.709084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.709098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.709186 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.709269 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.710347 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.711579 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.712426 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.713033 (+ 607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.713123 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.713148 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.713192 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260925 02:22:53.713271 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.708306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.11.125.2:36579)
0925 02:22:53.709018 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.709023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.709036 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:53.709080 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.709082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.709098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.709354 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.709419 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.710473 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.711742 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.712521 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.712591 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.712719 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.712746 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.712803 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":113,"threads_started":1}
I20260925 02:22:53.713531 18851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97317237fd34a08b77fb2d18f142f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:53.713881 18989 tablet_service.cc:1467] Processing CreateTablet for tablet e97317237fd34a08b77fb2d18f142f79 (DEFAULT_TABLE table=test-table [id=2470e14a7ff64c53afdfeebcbff12e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:53.713958 19126 tablet_service.cc:1467] Processing CreateTablet for tablet e97317237fd34a08b77fb2d18f142f79 (DEFAULT_TABLE table=test-table [id=2470e14a7ff64c53afdfeebcbff12e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:53.714119 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97317237fd34a08b77fb2d18f142f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:53.714270 19126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97317237fd34a08b77fb2d18f142f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:53.716830 19218 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Bootstrap starting.
I20260925 02:22:53.717324 19219 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Bootstrap starting.
I20260925 02:22:53.717549 19218 tablet_bootstrap.cc:654] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:53.717756 19220 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Bootstrap starting.
I20260925 02:22:53.717969 19219 tablet_bootstrap.cc:654] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:53.718086 19218 log.cc:824] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:53.718349 19220 tablet_bootstrap.cc:654] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:53.718685 19220 log.cc:824] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:53.718878 19219 log.cc:824] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:53.719149 19218 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: No bootstrap required, opened a new log
I20260925 02:22:53.719224 19218 ts_tablet_manager.cc:1397] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:53.719647 19220 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: No bootstrap required, opened a new log
I20260925 02:22:53.719726 19220 ts_tablet_manager.cc:1397] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:53.721560 19219 tablet_bootstrap.cc:492] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: No bootstrap required, opened a new log
I20260925 02:22:53.721494 19218 raft_consensus.cc:348] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.721652 19218 raft_consensus.cc:374] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:53.721686 19219 ts_tablet_manager.cc:1397] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:22:53.721689 19218 raft_consensus.cc:729] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3477e3986f4e72a4650574cdffef86, State: Initialized, Role: FOLLOWER
I20260925 02:22:53.721833 19218 consensus_queue.cc:260] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.722270 19218 ts_tablet_manager.cc:1428] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 02:22:53.722414 19056 heartbeater.cc:503] Master 127.11.125.62:44185 was elected leader, sending a full tablet report...
I20260925 02:22:53.722512 19220 raft_consensus.cc:348] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.722664 19220 raft_consensus.cc:374] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:53.722704 19220 raft_consensus.cc:729] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d419d3ab4c460d8fefc7763a297006, State: Initialized, Role: FOLLOWER
I20260925 02:22:53.722828 19220 consensus_queue.cc:260] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.723155 19220 ts_tablet_manager.cc:1428] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:22:53.723280 19193 heartbeater.cc:503] Master 127.11.125.62:44185 was elected leader, sending a full tablet report...
I20260925 02:22:53.726990 19219 raft_consensus.cc:348] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.727113 19219 raft_consensus.cc:374] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:53.727133 19219 raft_consensus.cc:729] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d93e4ae60d4eb894e381f753b6f0a3, State: Initialized, Role: FOLLOWER
I20260925 02:22:53.727205 19219 consensus_queue.cc:260] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.727458 19219 ts_tablet_manager.cc:1428] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:22:53.727582 18918 heartbeater.cc:503] Master 127.11.125.62:44185 was elected leader, sending a full tablet report...
W20260925 02:22:53.794188 19059 tablet.cc:2369] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:53.807761 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.805233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39856)
0925 02:22:53.805312 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.805324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.805372 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.805540 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.805540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.805542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.805596 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.805597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.806289 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.806289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.806985 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.806987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.807546 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.807547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.807593 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.807596 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.807635 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.807675 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:22:53.807751 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.805345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.11.125.60:34755)
0925 02:22:53.805406 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.805410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.805422 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:53.805450 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.805450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.805469 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.805526 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.805655 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.806213 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.807060 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.807472 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.807530 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.807590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.807649 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.807678 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:22:53.808233 18748 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 02:22:53.808684 18777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c42f340228483e95825a21b8b6c815. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.808808 18777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:22:53.838459 19226 raft_consensus.cc:482] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:53.838598 19226 raft_consensus.cc:504] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.839118 19226 leader_election.cc:288] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579), 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:53.843148 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.839406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:36797 (local address 127.11.125.3:37381)
0925 02:22:53.839466 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.839468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.839481 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:22:53.839514 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.839515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.839525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.839600 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.839898 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.840650 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.842174 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.842780 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.842892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.842902 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.843060 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.843096 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:53.843197 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.839312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:37381 (local address 127.11.125.1:36797)
0925 02:22:53.839396 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.839409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.839455 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.839618 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.839619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.839621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.839731 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.839732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.840742 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.840743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.842075 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.842078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.842895 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.842895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.842952 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.842956 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.843030 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.843090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":89}
I20260925 02:22:53.843443 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.839313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36579 (local address 127.11.125.1:46237)
0925 02:22:53.839396 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:53.839409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:53.839455 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:53.839618 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:53.839619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:53.839621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:53.839799 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.839801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.840741 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.840743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.842074 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:53.842078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.843011 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.843012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:53.843175 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.843178 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:53.843235 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:22:53.843274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":34}
I20260925 02:22:53.843508 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.839409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:46237 (local address 127.11.125.2:36579)
0925 02:22:53.839466 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.839469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.839481 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:53.839514 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.839515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.839525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.839600 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:53.839884 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.840650 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.842180 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.842938 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.843004 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.843029 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.843390 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.843432 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:22:53.843595 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97317237fd34a08b77fb2d18f142f79" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d419d3ab4c460d8fefc7763a297006" is_pre_election: true
I20260925 02:22:53.843812 19146 raft_consensus.cc:2454] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 0.
I20260925 02:22:53.843871 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97317237fd34a08b77fb2d18f142f79" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86" is_pre_election: true
I20260925 02:22:53.844046 19010 raft_consensus.cc:2454] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 0.
I20260925 02:22:53.844099 18806 leader_election.cc:302] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d93e4ae60d4eb894e381f753b6f0a3, 45d419d3ab4c460d8fefc7763a297006; no voters:
I20260925 02:22:53.844241 19226 raft_consensus.cc:2781] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:53.844295 19226 raft_consensus.cc:482] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:53.844327 19226 raft_consensus.cc:3037] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:53.845280 19226 raft_consensus.cc:504] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.845436 19226 leader_election.cc:288] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579), 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381)
I20260925 02:22:53.845613 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97317237fd34a08b77fb2d18f142f79" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d419d3ab4c460d8fefc7763a297006"
I20260925 02:22:53.845613 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97317237fd34a08b77fb2d18f142f79" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86"
I20260925 02:22:53.845693 19146 raft_consensus.cc:3037] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:53.845693 19010 raft_consensus.cc:3037] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:53.847035 19010 raft_consensus.cc:2454] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 1.
I20260925 02:22:53.847234 19146 raft_consensus.cc:2454] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 1.
I20260925 02:22:53.847244 18807 leader_election.cc:302] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3477e3986f4e72a4650574cdffef86, 20d93e4ae60d4eb894e381f753b6f0a3; no voters:
I20260925 02:22:53.847430 19226 raft_consensus.cc:2781] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:53.847608 19226 raft_consensus.cc:686] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 20d93e4ae60d4eb894e381f753b6f0a3, State: Running, Role: LEADER
I20260925 02:22:53.847719 19226 consensus_queue.cc:237] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } }
I20260925 02:22:53.848871 17939 catalog_manager.cc:5495] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:53.849575 18700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: NON_VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: true } } }
I20260925 02:22:53.849664 18700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.850862 18503 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: attempting to promote NON_VOTER 3cbeff6304a0443092aa55c7cb732f24 to VOTER
I20260925 02:22:53.851053 18503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:53.851871 18550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 02:22:53.852161 19234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 02:22:53.852881 18748 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 LEARNER]: Refusing update from remote peer 13c42f340228483e95825a21b8b6c815: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 02:22:53.852893 19232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.853065 18550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.853367 19235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 02:22:53.853781 18748 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.854056 19232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.854143 19232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.854089 19235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.854246 19235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.854938 18777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.855011 18777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:22:53.855434 19236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.855540 19236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:22:53.858364 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.856031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.11.125.1:33431)
0925 02:22:53.856069 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.856071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.856077 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:22:53.856106 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.856107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.856123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.856173 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:53.856290 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.856738 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.857691 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.857886 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.857940 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.858079 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:53.858175 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:53.858230 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.858260 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.858283 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:22:53.860296 19146 raft_consensus.cc:1261] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Refusing update from remote peer 20d93e4ae60d4eb894e381f753b6f0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:53.860646 19226 consensus_queue.cc:1035] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:53.860749 19010 raft_consensus.cc:1261] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Refusing update from remote peer 20d93e4ae60d4eb894e381f753b6f0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:53.860986 19226 consensus_queue.cc:1035] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:53.864604 19238 mvcc.cc:205] Tried to move back new op lower bound from 7333080980928851968 to 7333080980880760832. Current Snapshot: MvccSnapshot[applied={T|T < 7333080980928851968 or (T in {7333080980928851968})}]
I20260925 02:22:53.864948 19239 mvcc.cc:205] Tried to move back new op lower bound from 7333080980928851968 to 7333080980880760832. Current Snapshot: MvccSnapshot[applied={T|T < 7333080980928851968}]
I20260925 02:22:53.866667 19240 mvcc.cc:205] Tried to move back new op lower bound from 7333080980928851968 to 7333080980880760832. Current Snapshot: MvccSnapshot[applied={T|T < 7333080980928851968}]
I20260925 02:22:53.866839 17939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:53.866918 17939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:53.867650 19235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c42f340228483e95825a21b8b6c815" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:53.867745 19235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:53.868995 17939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13c42f340228483e95825a21b8b6c815: Sending DeleteTablet for 3 replicas of tablet e97317237fd34a08b77fb2d18f142f79
I20260925 02:22:53.869251 18851 tablet_service.cc:1514] Processing DeleteTablet for tablet e97317237fd34a08b77fb2d18f142f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:53 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:42950
I20260925 02:22:53.869496 19247 tablet_replica.cc:330] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: stopping tablet replica
I20260925 02:22:53.869473 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet e97317237fd34a08b77fb2d18f142f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:53 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:42950
I20260925 02:22:53.869611 19247 raft_consensus.cc:2229] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:53.869613 19126 tablet_service.cc:1514] Processing DeleteTablet for tablet e97317237fd34a08b77fb2d18f142f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:53 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:56428
I20260925 02:22:53.869729 19247 raft_consensus.cc:2258] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:53.870177 19247 ts_tablet_manager.cc:1905] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:53.870157 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet e97317237fd34a08b77fb2d18f142f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:53 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:44030
I20260925 02:22:53.870468 19249 tablet_replica.cc:330] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: stopping tablet replica
I20260925 02:22:53.870602 19249 raft_consensus.cc:2229] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:53.870739 19249 raft_consensus.cc:2258] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:53.871713 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 17908
I20260925 02:22:53.872556 19247 ts_tablet_manager.cc:1918] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:53.872645 19247 log.cc:1196] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/e97317237fd34a08b77fb2d18f142f79
I20260925 02:22:53.872896 19247 ts_tablet_manager.cc:1939] T e97317237fd34a08b77fb2d18f142f79 P 20d93e4ae60d4eb894e381f753b6f0a3: Deleting consensus metadata
W20260925 02:22:53.880154 18806 connection.cc:535] server connection from 127.0.0.1:42950 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:22:53.880539 19249 ts_tablet_manager.cc:1905] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:53.882357 19249 ts_tablet_manager.cc:1918] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:53.882428 19249 log.cc:1196] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/e97317237fd34a08b77fb2d18f142f79
I20260925 02:22:53.882710 19249 ts_tablet_manager.cc:1939] T e97317237fd34a08b77fb2d18f142f79 P 1b3477e3986f4e72a4650574cdffef86: Deleting consensus metadata
W20260925 02:22:53.883074 18942 connection.cc:441] server connection from 127.0.0.1:44030 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44030 (request call id 1) could send its response
I20260925 02:22:53.883472 19248 tablet_replica.cc:330] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: stopping tablet replica
I20260925 02:22:53.883587 19248 raft_consensus.cc:2229] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:53.883703 19248 raft_consensus.cc:2258] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:53.884169 19248 ts_tablet_manager.cc:1905] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:53.885964 19248 ts_tablet_manager.cc:1918] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:53.886081 19248 log.cc:1196] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/e97317237fd34a08b77fb2d18f142f79
I20260925 02:22:53.886389 19248 ts_tablet_manager.cc:1939] T e97317237fd34a08b77fb2d18f142f79 P 45d419d3ab4c460d8fefc7763a297006: Deleting consensus metadata
W20260925 02:22:53.886922 19079 connection.cc:441] server connection from 127.0.0.1:56428 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56428 (request call id 1) could send its response
I20260925 02:22:53.890077 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.871475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.11.125.60:34755)
0925 02:22:53.871517 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.871519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.871527 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:53.871547 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.871548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.871558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.871620 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.881677 (+ 10057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.882429 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.883860 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.884148 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.884196 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.885934 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.885936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.885936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.885943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.887919 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.888598 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.888599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.888600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.888656 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.888891 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.888893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.888893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.889312 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.889396 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.889616 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.889695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:53.892586 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.882110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.11.125.61:44137)
0925 02:22:53.882153 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.882155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.882868 (+ 713us) server_negotiation.cc:407] Connection header received
0925 02:22:53.882965 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.882966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.882981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.883010 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.885076 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.885826 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.887639 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.887978 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.888019 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.889500 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.889502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.889503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.889510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.891368 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.891792 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.891793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.891793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.891858 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.892084 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.892086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.892086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.892208 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.892314 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.892416 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.892492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:53.892971 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:53.882463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.11.125.60:34755)
0925 02:22:53.882503 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:53.882505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:53.883627 (+ 1122us) server_negotiation.cc:407] Connection header received
0925 02:22:53.883732 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:53.883732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:53.883742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:53.883772 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:53.886479 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.887199 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.888357 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:53.888652 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:53.888692 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:53.890095 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:53.890096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:53.890097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:53.890103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:53.891650 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.892029 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.892030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.892030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.892083 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:53.892304 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:53.892305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:53.892305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:53.892425 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:53.892491 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:53.892785 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:22:53.892860 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:54.866027 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:54.865716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:37279)
0925 02:22:54.865862 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:54.865954 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260925 02:22:54.866272 19193 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44185 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:54.869879 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:54.869655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:40579)
0925 02:22:54.869768 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:54.869840 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260925 02:22:54.870095 18918 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44185 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:54.870503 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:54.870281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:60451)
0925 02:22:54.870385 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:54.870448 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
W20260925 02:22:54.870734 19056 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44185 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.368808 19261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13c42f340228483e95825a21b8b6c815)
I20260925 02:22:55.368945 19262 raft_consensus.cc:482] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 13c42f340228483e95825a21b8b6c815)
I20260925 02:22:55.369113 19262 raft_consensus.cc:504] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:55.369112 19261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:55.369426 19261 leader_election.cc:288] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61:44137)
I20260925 02:22:55.369625 19262 leader_election.cc:288] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60:34755)
I20260925 02:22:55.369755 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.369545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60794)
0925 02:22:55.369657 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.369717 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260925 02:22:55.369867 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.369583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60796)
0925 02:22:55.369744 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.369793 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":44,"threads_started":1}
W20260925 02:22:55.369867 18528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
W20260925 02:22:55.369982 18724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.370512 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.370422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60812)
0925 02:22:55.370460 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.370485 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 02:22:55.370586 18528 leader_election.cc:334] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.370607 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.370497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60826)
0925 02:22:55.370547 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.370573 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:22:55.370668 18724 leader_election.cc:334] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.373382 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.369554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:34755 (local address 127.0.0.1:39874)
0925 02:22:55.369959 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.369969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:55.370061 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:55.370296 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:55.370297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:55.370300 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:55.370358 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:55.370360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.371235 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.371236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:55.372422 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:55.372425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.373151 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.373152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:55.373202 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:55.373206 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:55.373251 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:22:55.373294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":38,"threads_started":1}
I20260925 02:22:55.373394 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.369696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.11.125.60:34755)
0925 02:22:55.369960 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:55.369962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:55.370030 (+ 68us) server_negotiation.cc:407] Connection header received
0925 02:22:55.370140 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:55.370141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:55.370153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:55.370194 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:55.370445 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.371153 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.372509 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.373073 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.373148 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:55.373159 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:55.373278 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 02:22:55.373328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":41,"threads_started":1}
I20260925 02:22:55.373589 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.369731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:44137 (local address 127.0.0.1:39638)
0925 02:22:55.369822 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.369833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:55.369889 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:55.370473 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:55.370475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:55.370477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:55.370533 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:55.370536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.371480 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.371481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:55.372595 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:55.372597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.373346 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.373348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:55.373413 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:55.373418 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:55.373475 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:22:55.373525 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":43,"threads_started":1}
I20260925 02:22:55.373656 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.370147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.11.125.61:44137)
0925 02:22:55.370317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:55.370319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:55.370327 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:55.370355 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:55.370355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:55.370367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:55.370838 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:55.370872 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.371402 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.372672 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:55.373257 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:55.373331 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:55.373341 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:55.373549 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:22:55.373590 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260925 02:22:55.373608 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3cbeff6304a0443092aa55c7cb732f24" is_pre_election: true
I20260925 02:22:55.373705 18748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e241c2dea25b48e19aa47a8ec5d09e78 in term 1.
I20260925 02:22:55.373891 18526 leader_election.cc:302] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cbeff6304a0443092aa55c7cb732f24, e241c2dea25b48e19aa47a8ec5d09e78; no voters: 13c42f340228483e95825a21b8b6c815
I20260925 02:22:55.373910 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cbeff6304a0443092aa55c7cb732f24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" is_pre_election: true
I20260925 02:22:55.374037 19262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:22:55.374080 18550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbeff6304a0443092aa55c7cb732f24 in term 1.
I20260925 02:22:55.374122 19262 raft_consensus.cc:482] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 13c42f340228483e95825a21b8b6c815)
I20260925 02:22:55.374158 19262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:22:55.374253 18724 leader_election.cc:302] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cbeff6304a0443092aa55c7cb732f24, e241c2dea25b48e19aa47a8ec5d09e78; no voters: 13c42f340228483e95825a21b8b6c815
I20260925 02:22:55.374352 19261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:22:55.374392 19261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 13c42f340228483e95825a21b8b6c815)
I20260925 02:22:55.374411 19261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:22:55.375005 19262 raft_consensus.cc:504] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:55.375093 19261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:55.375165 19262 leader_election.cc:288] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 election: Requested vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), 3cbeff6304a0443092aa55c7cb732f24 (127.11.125.60:34755)
I20260925 02:22:55.375231 19261 leader_election.cc:288] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 election: Requested vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61:44137)
I20260925 02:22:55.375356 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3cbeff6304a0443092aa55c7cb732f24"
I20260925 02:22:55.375440 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.375306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60842)
0925 02:22:55.375394 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.375419 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 02:22:55.375428 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cbeff6304a0443092aa55c7cb732f24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
I20260925 02:22:55.375448 18748 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e241c2dea25b48e19aa47a8ec5d09e78 in current term 2: Already voted for candidate 3cbeff6304a0443092aa55c7cb732f24 in this term.
I20260925 02:22:55.375505 18550 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cbeff6304a0443092aa55c7cb732f24 in current term 2: Already voted for candidate e241c2dea25b48e19aa47a8ec5d09e78 in this term.
I20260925 02:22:55.375669 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.375236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60836)
0925 02:22:55.375351 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.375382 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 02:22:55.375773 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.375676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60848)
0925 02:22:55.375711 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.375742 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 02:22:55.375945 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.375869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:60850)
0925 02:22:55.375906 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.375930 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 02:22:55.375944 18724 leader_election.cc:334] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.376005 18724 leader_election.cc:302] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cbeff6304a0443092aa55c7cb732f24; no voters: 13c42f340228483e95825a21b8b6c815, e241c2dea25b48e19aa47a8ec5d09e78
W20260925 02:22:55.376020 18528 leader_election.cc:334] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:55.376055 18528 leader_election.cc:302] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e241c2dea25b48e19aa47a8ec5d09e78; no voters: 13c42f340228483e95825a21b8b6c815, 3cbeff6304a0443092aa55c7cb732f24
I20260925 02:22:55.376111 19261 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:22:55.376183 19262 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:22:55.866909 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.866636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:39829)
0925 02:22:55.866768 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.866843 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260925 02:22:55.870609 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.870410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:51275)
0925 02:22:55.870528 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.870568 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260925 02:22:55.871236 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:55.871062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:43217)
0925 02:22:55.871167 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:55.871200 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":46,"threads_started":1}
I20260925 02:22:56.867631 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.867388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:47391)
0925 02:22:56.867522 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.867569 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260925 02:22:56.871179 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.870997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:56007)
0925 02:22:56.871103 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.871140 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260925 02:22:56.871878 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.871661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:58925)
0925 02:22:56.871801 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.871840 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":49,"threads_started":1}
I20260925 02:22:56.970240 19277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:56.970427 19277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:56.970702 19277 leader_election.cc:288] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61:44137)
I20260925 02:22:56.971107 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.970864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32834)
0925 02:22:56.971001 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.971055 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20260925 02:22:56.971159 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cbeff6304a0443092aa55c7cb732f24" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" is_pre_election: true
I20260925 02:22:56.971328 18550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbeff6304a0443092aa55c7cb732f24 in term 2.
I20260925 02:22:56.971581 18724 leader_election.cc:302] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbeff6304a0443092aa55c7cb732f24, e241c2dea25b48e19aa47a8ec5d09e78; no voters:
I20260925 02:22:56.971630 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.971524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32838)
0925 02:22:56.971580 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.971608 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:22:56.971706 19277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260925 02:22:56.971746 18724 leader_election.cc:334] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:56.971765 19277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:56.971797 19277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:22:56.972537 19277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:56.972688 19277 leader_election.cc:288] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 election: Requested vote from peers 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185), e241c2dea25b48e19aa47a8ec5d09e78 (127.11.125.61:44137)
I20260925 02:22:56.972863 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.972773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32850)
0925 02:22:56.972808 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.972838 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:22:56.972983 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cbeff6304a0443092aa55c7cb732f24" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e241c2dea25b48e19aa47a8ec5d09e78"
I20260925 02:22:56.973096 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:22:56.973233 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.973106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32862)
0925 02:22:56.973164 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.973201 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:22:56.973349 18724 leader_election.cc:334] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
I20260925 02:22:56.973914 18550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbeff6304a0443092aa55c7cb732f24 in term 3.
I20260925 02:22:56.974112 18724 leader_election.cc:302] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cbeff6304a0443092aa55c7cb732f24, e241c2dea25b48e19aa47a8ec5d09e78; no voters: 13c42f340228483e95825a21b8b6c815
I20260925 02:22:56.974222 19277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:22:56.974347 19277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [term 3 LEADER]: Becoming Leader. State: Replica: 3cbeff6304a0443092aa55c7cb732f24, State: Running, Role: LEADER
I20260925 02:22:56.974466 19277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } }
I20260925 02:22:56.974926 19280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cbeff6304a0443092aa55c7cb732f24. Latest consensus state: current_term: 3 leader_uuid: "3cbeff6304a0443092aa55c7cb732f24" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:56.975028 19280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:56.975243 19282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:56.975685 19282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:56.975805 19282 catalog_manager.cc:1259] Loaded cluster ID: 1983e339a3484d1aaffef1c0dab67afb
I20260925 02:22:56.975857 19282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:56.976052 19282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:56.976152 19282 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Loaded TSK: 0
I20260925 02:22:56.976325 19282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:22:56.981630 18737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:56.983420 18737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:56.984421 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.984321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32876)
0925 02:22:56.984371 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.984399 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:22:56.984546 18550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [term 3 FOLLOWER]: Refusing update from remote peer 3cbeff6304a0443092aa55c7cb732f24: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 02:22:56.984824 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.984703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.0.0.1:32886)
0925 02:22:56.984752 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.984780 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:22:56.984830 19277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 02:22:56.985039 18724 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 -> Peer 13c42f340228483e95825a21b8b6c815 (127.11.125.62:44185): Couldn't send request to peer 13c42f340228483e95825a21b8b6c815. Status: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:22:56.985508 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cbeff6304a0443092aa55c7cb732f24. Latest consensus state: current_term: 3 leader_uuid: "3cbeff6304a0443092aa55c7cb732f24" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:56.985622 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:22:56.986030 19280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3cbeff6304a0443092aa55c7cb732f24" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:56.986099 19285 mvcc.cc:205] Tried to move back new op lower bound from 7333080993726296064 to 7333080993688453120. Current Snapshot: MvccSnapshot[applied={T|T < 7333080993726296064}]
I20260925 02:22:56.986207 19280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:56.986385 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3cbeff6304a0443092aa55c7cb732f24" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "13c42f340228483e95825a21b8b6c815" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44185 } } peers { permanent_uuid: "e241c2dea25b48e19aa47a8ec5d09e78" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 44137 } attrs { promote: false } } peers { permanent_uuid: "3cbeff6304a0443092aa55c7cb732f24" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 34755 } attrs { promote: false } } }
I20260925 02:22:56.986486 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P e241c2dea25b48e19aa47a8ec5d09e78 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:22:56.991145 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.987746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:37381 (local address 127.0.0.1:56444)
0925 02:22:56.987815 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.987826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:56.987878 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:56.988349 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:56.988350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:56.988351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:56.988411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.988412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989527 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.989529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.990417 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.990420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.990984 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.990984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.991026 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.991030 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:56.991067 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 02:22:56.991100 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:22:56.991183 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.987880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.11.125.3:37381)
0925 02:22:56.988130 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:56.988132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:56.988140 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:56.988163 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:56.988164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:56.988173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:56.988322 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:56.988506 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989353 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.990516 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.990918 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.990984 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.990998 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:56.991072 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:22:56.991109 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":54,"threads_started":1}
I20260925 02:22:56.991554 19126 tablet_service.cc:1467] Processing CreateTablet for tablet a35c2ea87079438db9ffef675318b5b0 (DEFAULT_TABLE table=test-table [id=acc5e6ed9b8f446088d3944a215a0cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:56.991708 19126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35c2ea87079438db9ffef675318b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:56.991916 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.987745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:33431 (local address 127.0.0.1:42966)
0925 02:22:56.988402 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.988411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:56.988494 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:56.988667 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:56.988668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:56.988670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:56.988729 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.988730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989568 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.989569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.990823 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.990825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.991679 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.991680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.991738 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.991760 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:22:56.991824 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:22:56.991871 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":212,"thread_start_us":81,"threads_started":1}
I20260925 02:22:56.992154 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.987928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.11.125.1:33431)
0925 02:22:56.988045 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:56.988047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:56.988464 (+ 417us) server_negotiation.cc:407] Connection header received
0925 02:22:56.988555 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:56.988556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:56.988567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:56.988660 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:56.988849 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989467 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.990920 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.991595 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.991668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.991683 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:56.992032 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:22:56.992083 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260925 02:22:56.992795 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.987825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36579 (local address 127.0.0.1:44040)
0925 02:22:56.988296 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.988306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:56.988364 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:56.988568 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:56.988569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:56.988570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:56.988600 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.988602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989630 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.989631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.990822 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:56.990825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.992567 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.992569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:56.992637 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.992641 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:56.992693 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:22:56.992731 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":26,"thread_start_us":74,"threads_started":1}
I20260925 02:22:56.993040 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.988074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.11.125.2:36579)
0925 02:22:56.988231 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:56.988233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:56.988416 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:22:56.988447 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:56.988448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:56.988460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:56.988527 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:56.988667 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.989543 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.991722 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:56.992372 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:56.992522 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:56.992537 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:56.992892 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 02:22:56.992961 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260925 02:22:56.993273 18989 tablet_service.cc:1467] Processing CreateTablet for tablet a35c2ea87079438db9ffef675318b5b0 (DEFAULT_TABLE table=test-table [id=acc5e6ed9b8f446088d3944a215a0cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:56.993336 19294 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Bootstrap starting.
I20260925 02:22:56.993438 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35c2ea87079438db9ffef675318b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:56.993443 18850 tablet_service.cc:1467] Processing CreateTablet for tablet a35c2ea87079438db9ffef675318b5b0 (DEFAULT_TABLE table=test-table [id=acc5e6ed9b8f446088d3944a215a0cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:56.993551 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35c2ea87079438db9ffef675318b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:56.993722 19294 tablet_bootstrap.cc:654] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:56.994356 19294 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: No bootstrap required, opened a new log
I20260925 02:22:56.994403 19294 ts_tablet_manager.cc:1397] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:56.994522 19294 raft_consensus.cc:348] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.994581 19294 raft_consensus.cc:374] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:56.994599 19294 raft_consensus.cc:729] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d419d3ab4c460d8fefc7763a297006, State: Initialized, Role: FOLLOWER
I20260925 02:22:56.994627 19294 consensus_queue.cc:260] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.994828 19294 ts_tablet_manager.cc:1428] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:56.994959 19298 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Bootstrap starting.
I20260925 02:22:56.995102 19297 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: Bootstrap starting.
I20260925 02:22:56.995230 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.995063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:48831)
0925 02:22:56.995156 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.995183 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:22:56.995378 19298 tablet_bootstrap.cc:654] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:56.995515 19297 tablet_bootstrap.cc:654] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:56.995954 19298 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: No bootstrap required, opened a new log
I20260925 02:22:56.996002 19298 ts_tablet_manager.cc:1397] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:56.996174 19297 tablet_bootstrap.cc:492] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: No bootstrap required, opened a new log
I20260925 02:22:56.996157 19298 raft_consensus.cc:348] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.996227 19297 ts_tablet_manager.cc:1397] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:56.996234 19298 raft_consensus.cc:374] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:56.996273 19298 raft_consensus.cc:729] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3477e3986f4e72a4650574cdffef86, State: Initialized, Role: FOLLOWER
I20260925 02:22:56.996341 19298 consensus_queue.cc:260] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.996341 19297 raft_consensus.cc:348] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.996403 19297 raft_consensus.cc:374] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:56.996438 19297 raft_consensus.cc:729] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d93e4ae60d4eb894e381f753b6f0a3, State: Initialized, Role: FOLLOWER
I20260925 02:22:56.996497 19297 consensus_queue.cc:260] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:56.996558 19298 ts_tablet_manager.cc:1428] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:56.996714 19297 ts_tablet_manager.cc:1428] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:56.996809 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.996697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:45553)
0925 02:22:56.996735 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.996763 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 02:22:56.997212 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.997114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:52553)
0925 02:22:56.997155 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.997188 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 02:22:56.997478 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:56.997377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:52367)
0925 02:22:56.997418 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:56.997450 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 02:22:56.997553 18919 heartbeater.cc:503] Master 127.11.125.60:34755 was elected leader, sending a full tablet report...
I20260925 02:22:56.997581 19057 heartbeater.cc:503] Master 127.11.125.60:34755 was elected leader, sending a full tablet report...
W20260925 02:22:57.145334 18921 tablet.cc:2369] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:22:57.186396 19196 tablet.cc:2369] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:57.371181 19296 raft_consensus.cc:482] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:57.371313 19296 raft_consensus.cc:504] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:57.371629 19296 leader_election.cc:288] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431), 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:57.375242 19302 raft_consensus.cc:482] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:57.375355 19302 raft_consensus.cc:504] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:57.375536 19302 leader_election.cc:288] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381), 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:57.375747 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.371831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:33431 (local address 127.11.125.3:51167)
0925 02:22:57.371903 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.371917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:57.371999 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:57.372152 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:57.372153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:57.372156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:57.372260 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:57.372262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.373253 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.373253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:57.374689 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:57.374692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.375466 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.375466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:57.375575 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:57.375579 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:57.375628 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:57.375669 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":83}
I20260925 02:22:57.375748 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86" is_pre_election: true
I20260925 02:22:57.375874 19010 raft_consensus.cc:2454] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 0.
I20260925 02:22:57.375744 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d419d3ab4c460d8fefc7763a297006" is_pre_election: true
I20260925 02:22:57.375912 19146 raft_consensus.cc:2454] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 0.
I20260925 02:22:57.375988 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.371933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:51167 (local address 127.11.125.1:33431)
0925 02:22:57.372007 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:57.372009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:57.372023 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:22:57.372064 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:57.372065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:57.372074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:57.372124 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:57.372356 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.373170 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.374795 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.375354 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.375416 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:57.375424 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:57.375880 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 02:22:57.375932 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:22:57.376019 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.371925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:45489 (local address 127.11.125.2:36579)
0925 02:22:57.371994 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:57.371996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:57.372164 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:22:57.372199 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:57.372200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:57.372216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:57.372260 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:57.372407 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.373163 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.374784 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.375332 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.375410 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:57.375422 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:57.375913 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 02:22:57.375957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:22:57.376179 18806 leader_election.cc:302] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d93e4ae60d4eb894e381f753b6f0a3, 45d419d3ab4c460d8fefc7763a297006; no voters:
I20260925 02:22:57.376204 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.371831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36579 (local address 127.11.125.3:45489)
0925 02:22:57.372063 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.372073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:57.372153 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:57.372285 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:57.372286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:57.372287 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:57.372340 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:57.372342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.373250 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.373251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:57.374689 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:57.374692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.375461 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.375462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:57.375814 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:57.375820 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:22:57.376086 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 02:22:57.376132 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":403,"thread_start_us":113,"threads_started":1}
I20260925 02:22:57.376201 18872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "45d419d3ab4c460d8fefc7763a297006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" is_pre_election: true
I20260925 02:22:57.376334 19302 raft_consensus.cc:2781] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:57.376376 18872 raft_consensus.cc:2454] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d419d3ab4c460d8fefc7763a297006 in term 0.
I20260925 02:22:57.376410 19302 raft_consensus.cc:482] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:57.376432 19302 raft_consensus.cc:3037] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:57.376485 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "45d419d3ab4c460d8fefc7763a297006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86" is_pre_election: true
I20260925 02:22:57.376575 19010 raft_consensus.cc:2454] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d419d3ab4c460d8fefc7763a297006 in term 0.
I20260925 02:22:57.376561 19080 leader_election.cc:302] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d93e4ae60d4eb894e381f753b6f0a3, 45d419d3ab4c460d8fefc7763a297006; no voters:
I20260925 02:22:57.376696 19296 raft_consensus.cc:2781] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:57.376739 19296 raft_consensus.cc:482] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:57.376758 19296 raft_consensus.cc:3037] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:57.377408 19302 raft_consensus.cc:504] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:57.377559 19302 leader_election.cc:288] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 election: Requested vote from peers 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381), 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:57.377600 19296 raft_consensus.cc:504] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:57.377758 19296 leader_election.cc:288] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [CANDIDATE]: Term 1 election: Requested vote from peers 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431), 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579)
I20260925 02:22:57.377744 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86"
I20260925 02:22:57.377825 19010 raft_consensus.cc:3037] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:57.377826 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d419d3ab4c460d8fefc7763a297006"
I20260925 02:22:57.377889 19146 raft_consensus.cc:2379] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 20d93e4ae60d4eb894e381f753b6f0a3 in current term 1: Already voted for candidate 45d419d3ab4c460d8fefc7763a297006 in this term.
I20260925 02:22:57.377871 18872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "45d419d3ab4c460d8fefc7763a297006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d93e4ae60d4eb894e381f753b6f0a3"
I20260925 02:22:57.377946 18872 raft_consensus.cc:2379] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 45d419d3ab4c460d8fefc7763a297006 in current term 1: Already voted for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in this term.
I20260925 02:22:57.378113 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35c2ea87079438db9ffef675318b5b0" candidate_uuid: "45d419d3ab4c460d8fefc7763a297006" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3477e3986f4e72a4650574cdffef86"
I20260925 02:22:57.378346 19082 leader_election.cc:302] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45d419d3ab4c460d8fefc7763a297006; no voters: 1b3477e3986f4e72a4650574cdffef86, 20d93e4ae60d4eb894e381f753b6f0a3
I20260925 02:22:57.378454 19296 raft_consensus.cc:2726] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:22:57.378654 19010 raft_consensus.cc:2454] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d93e4ae60d4eb894e381f753b6f0a3 in term 1.
I20260925 02:22:57.378822 18807 leader_election.cc:302] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b3477e3986f4e72a4650574cdffef86, 20d93e4ae60d4eb894e381f753b6f0a3; no voters: 45d419d3ab4c460d8fefc7763a297006
I20260925 02:22:57.378935 19302 raft_consensus.cc:2781] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:57.379101 19302 raft_consensus.cc:686] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 20d93e4ae60d4eb894e381f753b6f0a3, State: Running, Role: LEADER
I20260925 02:22:57.379154 19302 consensus_queue.cc:237] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } }
I20260925 02:22:57.379698 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.379560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:47729)
0925 02:22:57.379639 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.379670 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 02:22:57.379812 18738 catalog_manager.cc:5495] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d93e4ae60d4eb894e381f753b6f0a3" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 33431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:57.432246 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.429898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.11.125.1:33431)
0925 02:22:57.429970 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:57.429973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:57.429984 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:22:57.430016 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:57.430018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:57.430030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:57.430101 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:57.430240 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.430721 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.431504 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:57.431728 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:57.431827 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:57.431965 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:57.432004 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:57.432064 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:57.432129 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:22:57.432152 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:22:57.433319 19146 raft_consensus.cc:1261] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Refusing update from remote peer 20d93e4ae60d4eb894e381f753b6f0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:57.433314 19010 raft_consensus.cc:1261] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Refusing update from remote peer 20d93e4ae60d4eb894e381f753b6f0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:57.433604 19304 consensus_queue.cc:1035] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d419d3ab4c460d8fefc7763a297006" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:57.433718 19302 consensus_queue.cc:1035] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3477e3986f4e72a4650574cdffef86" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:57.434054 19194 heartbeater.cc:503] Master 127.11.125.60:34755 was elected leader, sending a full tablet report...
I20260925 02:22:57.434648 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.434530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:43599)
0925 02:22:57.434576 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.434607 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 02:22:57.434909 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.434512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:37247)
0925 02:22:57.434842 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.434877 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260925 02:22:57.435750 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.435499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:57581)
0925 02:22:57.435690 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.435722 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 02:22:57.436735 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.436598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.2:59595)
0925 02:22:57.436653 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.436704 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:22:57.436756 18736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:57.436888 18736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:57.437100 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.437001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.1:54399)
0925 02:22:57.437038 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.437070 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 02:22:57.437438 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:57.436073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44185 (local address 127.11.125.3:36083)
0925 02:22:57.437368 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.437400 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1276}
I20260925 02:22:57.438374 18736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cbeff6304a0443092aa55c7cb732f24: Sending DeleteTablet for 3 replicas of tablet a35c2ea87079438db9ffef675318b5b0
I20260925 02:22:57.438750 19126 tablet_service.cc:1514] Processing DeleteTablet for tablet a35c2ea87079438db9ffef675318b5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:57 UTC) from {username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 02:22:57.438791 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet a35c2ea87079438db9ffef675318b5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:57 UTC) from {username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:42966
I20260925 02:22:57.438946 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet a35c2ea87079438db9ffef675318b5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:57 UTC) from {username='oryx', principal='oryx/127.11.125.60@KRBTEST.COM'} at 127.0.0.1:44040
I20260925 02:22:57.439016 19317 tablet_replica.cc:330] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3: stopping tablet replica
I20260925 02:22:57.439061 19316 tablet_replica.cc:330] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: stopping tablet replica
I20260925 02:22:57.439085 19317 raft_consensus.cc:2229] T a35c2ea87079438db9ffef675318b5b0 P 20d93e4ae60d4eb894e381f753b6f0a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:57.439123 19316 raft_consensus.cc:2229] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:57.439195 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18690
I20260925 02:22:57.439232 19316 raft_consensus.cc:2258] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:57.439318 19318 tablet_replica.cc:330] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: stopping tablet replica
I20260925 02:22:57.439364 19318 raft_consensus.cc:2229] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:57.439404 19318 raft_consensus.cc:2258] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:57.440263 19316 ts_tablet_manager.cc:1905] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:57.442054 19316 ts_tablet_manager.cc:1918] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:57.442132 19316 log.cc:1196] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/a35c2ea87079438db9ffef675318b5b0
I20260925 02:22:57.442405 19318 ts_tablet_manager.cc:1905] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:57.442529 19316 ts_tablet_manager.cc:1939] T a35c2ea87079438db9ffef675318b5b0 P 45d419d3ab4c460d8fefc7763a297006: Deleting consensus metadata
I20260925 02:22:57.442867 18723 catalog_manager.cc:4841] TS 45d419d3ab4c460d8fefc7763a297006 (127.11.125.3:37381): tablet a35c2ea87079438db9ffef675318b5b0 (table test-table [id=acc5e6ed9b8f446088d3944a215a0cdd]) successfully deleted
I20260925 02:22:57.443673 19318 ts_tablet_manager.cc:1918] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:57.443727 19318 log.cc:1196] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/a35c2ea87079438db9ffef675318b5b0
I20260925 02:22:57.443964 19318 ts_tablet_manager.cc:1939] T a35c2ea87079438db9ffef675318b5b0 P 1b3477e3986f4e72a4650574cdffef86: Deleting consensus metadata
I20260925 02:22:57.444425 18724 catalog_manager.cc:4841] TS 1b3477e3986f4e72a4650574cdffef86 (127.11.125.2:36579): tablet a35c2ea87079438db9ffef675318b5b0 (table test-table [id=acc5e6ed9b8f446088d3944a215a0cdd]) successfully deleted
W20260925 02:22:57.447252 19292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:57.447092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:33431 (local address 127.0.0.1:42986)
0925 02:22:57.447156 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:57.447187 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:33431: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:22:57.447438 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18926
W20260925 02:22:57.447439 18722 catalog_manager.cc:4584] TS 20d93e4ae60d4eb894e381f753b6f0a3 (127.11.125.1:33431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35c2ea87079438db9ffef675318b5b0: Network error: Client connection negotiation failed: client connection to 127.11.125.1:33431: connect: Connection reset by peer (error 104)
I20260925 02:22:57.453191 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 19063
I20260925 02:22:57.458591 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18408
I20260925 02:22:57.463619 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 18596
2026-09-25T02:22:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6626 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:22:57 dist-test-slave-hj7s krb5kdc[19321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:22:57 dist-test-slave-hj7s krb5kdc[19321](info): set up 2 sockets
Sep 25 02:22:57 dist-test-slave-hj7s krb5kdc[19321](info): commencing operation
krb5kdc: starting...
W20260925 02:22:58.507756 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:22:58 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:22:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab.
Entry for principal oryx/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab.
I20260925 02:22:58.627564 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42533
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab
--principal=oryx/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:58.711808 19334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:58.712102 19334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:58.712143 19334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:58.713374 19334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:22:58.713450 19334 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:22:58.713475 19334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:58.713501 19334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:58.713533 19334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:22:58.713559 19334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:22:58.715121 19334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:42533
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.62.keytab
--principal=oryx/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:58.715417 19334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:58.715683 19334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:58.720731 19340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:58.720739 19342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:58.720739 19339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:58.721122 19334 server_base.cc:1034] running on GCE node
I20260925 02:22:58.721371 19334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:58.721616 19334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:58.722754 19334 hybrid_clock.cc:648] HybridClock initialized: now 1790302978722737 us; error 32 us; skew 500 ppm
Sep 25 02:22:58 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:58.729404 19334 init.cc:377] Logged in from keytab as oryx/127.11.125.62@KRBTEST.COM (short username oryx)
I20260925 02:22:58.729908 19334 webserver.cc:460] Webserver started at http://127.11.125.62:40923/ using document root <none> and password file <none>
I20260925 02:22:58.730154 19334 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:58.730222 19334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:58.730379 19334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:58.731498 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "9c64a62148c040d58d3ba6dd4de58242"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "7b8376f03ab45de783986bae1c80fbc9"
server_key_iv: "54c2ac758c35a96814c51c6dde2ef684"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.731873 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "9c64a62148c040d58d3ba6dd4de58242"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "7b8376f03ab45de783986bae1c80fbc9"
server_key_iv: "54c2ac758c35a96814c51c6dde2ef684"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.733259 19334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:58.734006 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:58.734184 19334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:58.734314 19334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "9c64a62148c040d58d3ba6dd4de58242"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "7b8376f03ab45de783986bae1c80fbc9"
server_key_iv: "54c2ac758c35a96814c51c6dde2ef684"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.734395 19334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:58.753055 19334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:58.754984 19334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:58.755162 19334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:58.760234 19334 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:42533
I20260925 02:22:58.760268 19412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:42533 every 8 connection(s)
I20260925 02:22:58.760635 19334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:22:58.761322 19413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:58.762647 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 19334
I20260925 02:22:58.762751 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:22:58.762955 11764 external_mini_cluster.cc:1428] Setting key 51a95cda109e77cda9b2418436aad1e3
I20260925 02:22:58.764207 19413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Bootstrap starting.
I20260925 02:22:58.764837 19413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:58.765247 19413 log.cc:824] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:58.766058 19413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: No bootstrap required, opened a new log
I20260925 02:22:58.768551 19413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } }
I20260925 02:22:58.768714 19413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:58.768801 19413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c64a62148c040d58d3ba6dd4de58242, State: Initialized, Role: FOLLOWER
I20260925 02:22:58.768977 19413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } }
I20260925 02:22:58.769076 19413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:22:58.769121 19413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:22:58.769170 19413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:58.769827 19413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } }
I20260925 02:22:58.769951 19413 leader_election.cc:302] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c64a62148c040d58d3ba6dd4de58242; no voters:
I20260925 02:22:58.770120 19413 leader_election.cc:288] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 25 02:22:58 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:58.770188 19418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:58.770402 19413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:22:58.770888 19418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [term 1 LEADER]: Becoming Leader. State: Replica: 9c64a62148c040d58d3ba6dd4de58242, State: Running, Role: LEADER
I20260925 02:22:58.771004 19418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } }
I20260925 02:22:58.771420 19418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c64a62148c040d58d3ba6dd4de58242. Latest consensus state: current_term: 1 leader_uuid: "9c64a62148c040d58d3ba6dd4de58242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } } }
I20260925 02:22:58.771536 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c64a62148c040d58d3ba6dd4de58242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64a62148c040d58d3ba6dd4de58242" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42533 } } }
I20260925 02:22:58.771577 19418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:58.771777 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242 [sys.catalog]: This master's current role is: LEADER
I20260925 02:22:58.771865 19428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:22:58.772643 19428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:22:58.774252 19428 catalog_manager.cc:1347] Generated new cluster ID: bdb94ea686464fac827ff0a2081966ea
I20260925 02:22:58.774307 19428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:22:58.775139 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:58.763536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.11.125.62:42533)
0925 02:22:58.763723 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:58.763731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:58.763757 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:22:58.764177 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:58.764184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:58.764231 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:58.764399 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:58.765092 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.766461 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.767209 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.767831 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.767948 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:58.770860 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:58.770865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:58.770865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:58.770877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:58.773079 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.773510 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.773514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.773514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.773617 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.773838 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.773838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.773839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.774127 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:58.774270 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:58.774796 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 02:22:58.774943 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260925 02:22:58.792341 19428 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:22:58.792987 19428 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:22:58.796386 19428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Generated new TSK 0
I20260925 02:22:58.796579 19428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
I20260925 02:22:58.824199 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42533
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:58.910147 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:58.910364 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:58.910391 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:58.911487 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:58.911521 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:58.911542 19440 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:58.911562 19440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:22:58.913056 19440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:58.913256 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:58.913448 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:58.918702 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:58.918676 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:58.918689 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:58.918866 19440 server_base.cc:1034] running on GCE node
I20260925 02:22:58.919133 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:58.919404 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:58.920665 19440 hybrid_clock.cc:648] HybridClock initialized: now 1790302978920646 us; error 33 us; skew 500 ppm
Sep 25 02:22:58 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:58.923245 19440 init.cc:377] Logged in from keytab as oryx/127.11.125.1@KRBTEST.COM (short username oryx)
I20260925 02:22:58.923748 19440 webserver.cc:460] Webserver started at http://127.11.125.1:42951/ using document root <none> and password file <none>
I20260925 02:22:58.923997 19440 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:58.924065 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:58.924217 19440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:58.925348 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "b1401078530e47e7a24c684135ffe3ff"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "29f49196f9bffa126390fead9d02191f"
server_key_iv: "9020648374cc35efa01d2226653d10ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.925716 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "b1401078530e47e7a24c684135ffe3ff"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "29f49196f9bffa126390fead9d02191f"
server_key_iv: "9020648374cc35efa01d2226653d10ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.927053 19440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:58.927840 19455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:58.928030 19440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:58.928140 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "b1401078530e47e7a24c684135ffe3ff"
format_stamp: "Formatted at 2026-09-25 02:22:58 on dist-test-slave-hj7s"
server_key: "29f49196f9bffa126390fead9d02191f"
server_key_iv: "9020648374cc35efa01d2226653d10ad"
server_key_version: "encryptionkey@0"
I20260925 02:22:58.928216 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:58.938194 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:58.939976 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:58.940141 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:58.940644 19440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:58.940709 19440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:58.940748 19440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:58.940799 19440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:22:58 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:58.947942 19440 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:39249
I20260925 02:22:58.948024 19572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:39249 every 8 connection(s)
I20260925 02:22:58.948355 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:22:58.949121 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 19440
I20260925 02:22:58.949234 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:22:58.949473 11764 external_mini_cluster.cc:1428] Setting key 03debbbcd395d03849bad487b7283335
I20260925 02:22:58.951416 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:58.941144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.1:42155)
0925 02:22:58.941307 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:58.941342 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:58.941860 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:58.942124 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:58.942127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:58.942463 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:58.942740 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:58.942748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.943843 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.943844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:58.945146 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:58.945149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.945745 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.945747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:58.945886 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:58.946402 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:58.946410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:58.947872 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:58.949940 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:58.949947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:58.949949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:58.950256 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:58.950507 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:58.950507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:58.950508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:58.950610 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:58.950916 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:58.950921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:58.951029 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 02:22:58.951110 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:22:58.951228 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260925 02:22:58.951411 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:58.941161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42155 (local address 127.11.125.62:42533)
0925 02:22:58.941204 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:58.941206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:58.941423 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:22:58.941967 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:58.941969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:58.941982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:58.942048 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:58.942863 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.943728 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.945253 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.945630 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.945717 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:58.947987 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:58.947988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:58.947988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:58.947997 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:58.949847 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.950368 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.950369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.950370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.950436 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.950675 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.950676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.950677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.950849 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:58.950920 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:58.951152 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:22:58.951255 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:58.952033 19573 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42533
I20260925 02:22:58.952185 19573 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:58.952445 19573 heartbeater.cc:511] Master 127.11.125.62:42533 requested a full tablet report, sending...
I20260925 02:22:58.953118 19365 ts_manager.cc:194] Registered new tserver with Master: b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249)
I20260925 02:22:58.954082 19365 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:42155
WARNING: no policy specified for oryx/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:22:58.959888 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:58.954309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:46217 (local address 127.11.125.62:42533)
0925 02:22:58.954372 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:58.954374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:58.954382 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:58.954400 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:58.954401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:58.954410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:58.954477 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:58.954873 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.955348 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.956058 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.956304 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.956401 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:58.957340 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:58.957342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:58.957342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:58.957349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:58.958697 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.959103 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.959105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.959105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.959166 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:58.959377 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:58.959378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:58.959378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:58.959486 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:58.959561 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:58.959719 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:22:58.959796 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:22:58.959939 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:58.953063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.1:46217)
0925 02:22:58.953181 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:58.953191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:58.953243 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:58.954498 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:58.954499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:58.954714 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:58.954791 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:58.954793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.955446 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.955447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:58.955971 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:58.955973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:58.956404 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:58.956405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:58.956471 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:58.956938 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:58.956943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:58.957230 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:58.958807 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:58.958808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:58.958809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:58.958984 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:58.959239 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:58.959240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:58.959241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:58.959307 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:58.959553 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:58.959554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:58.959601 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:22:58.959691 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:22:58.959802 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
I20260925 02:22:58.975909 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42533
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:59.057366 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:59.057556 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:59.057580 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:59.058665 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:59.058698 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:59.058718 19582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:59.058735 19582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:22:59.060252 19582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:59.060429 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:59.060601 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:59.065532 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:59.065524 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:59.065563 19582 server_base.cc:1034] running on GCE node
W20260925 02:22:59.065541 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:59.065968 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:59.066236 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:59.067400 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790302979067373 us; error 43 us; skew 500 ppm
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:59.069900 19582 init.cc:377] Logged in from keytab as oryx/127.11.125.2@KRBTEST.COM (short username oryx)
I20260925 02:22:59.070399 19582 webserver.cc:460] Webserver started at http://127.11.125.2:45217/ using document root <none> and password file <none>
I20260925 02:22:59.070601 19582 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:59.070641 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:59.070729 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:59.071828 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "0deabff8b4a54f28832359d889f105d0"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "b8778adb07bc38f34a77cae8bf801a44"
server_key_iv: "e0c467bb542451abf5cab1a3df275a30"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.072175 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "0deabff8b4a54f28832359d889f105d0"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "b8778adb07bc38f34a77cae8bf801a44"
server_key_iv: "e0c467bb542451abf5cab1a3df275a30"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.073506 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:22:59.074240 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.074430 19582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:59.074502 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "0deabff8b4a54f28832359d889f105d0"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "b8778adb07bc38f34a77cae8bf801a44"
server_key_iv: "e0c467bb542451abf5cab1a3df275a30"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.074553 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:59.091838 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:59.093739 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:59.093876 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:59.094422 19582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:59.094488 19582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.094537 19582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:59.094560 19582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:59.101745 19582 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:43509
I20260925 02:22:59.101832 19713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:43509 every 8 connection(s)
I20260925 02:22:59.102144 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:22:59.104854 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.094997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:42139 (local address 127.11.125.62:42533)
0925 02:22:59.095056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.095058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.095307 (+ 249us) server_negotiation.cc:407] Connection header received
0925 02:22:59.095877 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.095879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.095895 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.095976 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.096878 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.097751 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.098794 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.099120 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.099196 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.101863 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.101865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.101865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.101873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.103341 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.103710 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.103711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.103711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.103762 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.103998 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.103999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.103999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.104113 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.104165 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.104713 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.104789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:22:59.104998 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.094988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.2:42139)
0925 02:22:59.095168 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.095203 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.095721 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.096069 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.096074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.096441 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:59.096750 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.096757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.097885 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.097886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.098694 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.098696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.099236 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.099237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.099409 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.099927 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:59.099939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:59.101719 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:59.103417 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.103424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.103425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.103640 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.103843 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.103843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.103844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.103941 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.104182 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:59.104186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:59.104302 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.104618 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.104748 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260925 02:22:59.105561 19714 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42533
I20260925 02:22:59.105679 19714 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:59.105859 19714 heartbeater.cc:511] Master 127.11.125.62:42533 requested a full tablet report, sending...
I20260925 02:22:59.106355 19365 ts_manager.cc:194] Registered new tserver with Master: 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509)
I20260925 02:22:59.106824 19365 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:42139
I20260925 02:22:59.110867 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 19582
I20260925 02:22:59.110965 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:22:59.111162 11764 external_mini_cluster.cc:1428] Setting key 925da0f12d9612d9605de0c295aa306e
I20260925 02:22:59.112413 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.106560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.2:53335)
0925 02:22:59.106687 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.106697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.106748 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.107080 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.107081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.107298 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:59.107353 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.107355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.108106 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.108106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.108599 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.108600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.109029 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.109029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.109076 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.109536 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:59.109547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:59.109812 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:59.111310 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.111311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.111311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.111542 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.111731 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.111731 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.111732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.111802 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.112151 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:59.112152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:59.112189 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.112245 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.112344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260925 02:22:59.112527 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.106624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:53335 (local address 127.11.125.62:42533)
0925 02:22:59.106956 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.106958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.106967 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:22:59.106987 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.106987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.106997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.107051 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.107435 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.108012 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.108696 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.108893 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.109016 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.109906 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.109908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.109908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.109913 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.111217 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.111612 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.111613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.111613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.111664 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.111871 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.111871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.111872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.112073 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.112150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.112308 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.112390 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
WARNING: no policy specified for oryx/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
I20260925 02:22:59.138252 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42533
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:45317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:22:59.220623 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:22:59.220849 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:22:59.220875 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:22:59.222105 19723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:22:59.222149 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:22:59.222169 19723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:22:59.222229 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:22:59.223763 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:45317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.125.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:22:59.224026 19723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:22:59.224265 19723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:22:59.229388 19731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:59.229372 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:22:59.229372 19728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:22:59.229399 19723 server_base.cc:1034] running on GCE node
I20260925 02:22:59.229947 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:22:59.230216 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:22:59.231333 19723 hybrid_clock.cc:648] HybridClock initialized: now 1790302979231323 us; error 30 us; skew 500 ppm
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:22:59.234016 19723 init.cc:377] Logged in from keytab as oryx/127.11.125.3@KRBTEST.COM (short username oryx)
I20260925 02:22:59.234402 19723 webserver.cc:460] Webserver started at http://127.11.125.3:42631/ using document root <none> and password file <none>
I20260925 02:22:59.234593 19723 fs_manager.cc:359] Metadata directory not provided
I20260925 02:22:59.234633 19723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:22:59.234730 19723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:22:59.235735 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "1f0a1402f42f47bebfa49104d937e96f"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "669ecb3d275565adeec395e008d918ae"
server_key_iv: "c5b8dbd9383058633292d75bf1a6351b"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.236058 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "1f0a1402f42f47bebfa49104d937e96f"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "669ecb3d275565adeec395e008d918ae"
server_key_iv: "c5b8dbd9383058633292d75bf1a6351b"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.237397 19723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:22:59.238127 19738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.238296 19723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:59.238404 19723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "1f0a1402f42f47bebfa49104d937e96f"
format_stamp: "Formatted at 2026-09-25 02:22:59 on dist-test-slave-hj7s"
server_key: "669ecb3d275565adeec395e008d918ae"
server_key_iv: "c5b8dbd9383058633292d75bf1a6351b"
server_key_version: "encryptionkey@0"
I20260925 02:22:59.238481 19723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:22:59.251312 19723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:22:59.254091 19723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:22:59.254263 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:22:59.254832 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:22:59.254895 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.254938 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:22:59.254966 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.261997 19723 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:34729
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:59.262156 19854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:34729 every 8 connection(s)
I20260925 02:22:59.262413 19723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:22:59.263001 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 19723
I20260925 02:22:59.263124 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:22:59.263352 11764 external_mini_cluster.cc:1428] Setting key 4cb4e1170d7f4f87c4e9bfca22f33284
I20260925 02:22:59.265717 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.255300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:55059 (local address 127.11.125.62:42533)
0925 02:22:59.255344 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.255346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.255578 (+ 232us) server_negotiation.cc:407] Connection header received
0925 02:22:59.256117 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.256118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.256132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.256191 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.257071 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.257682 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.259330 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.259686 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.259778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.262710 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.262711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.262712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.262720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.264466 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.264850 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.264851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.264851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.264945 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.265171 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.265171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.265172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.265281 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.265334 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.265584 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.265648 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:59.265916 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.255301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.3:55059)
0925 02:22:59.255458 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.255492 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.256002 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.256268 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.256272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.256632 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:59.256962 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.256970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.257793 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.257794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.259209 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.259213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.259814 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.259815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.259958 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.260551 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:59.260560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:59.262608 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:59.264545 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.264552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.264554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.264770 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.265013 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.265014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.265014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.265116 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.265340 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:59.265345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:59.265472 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.265556 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.265679 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260925 02:22:59.266464 19855 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42533
I20260925 02:22:59.266546 19855 heartbeater.cc:464] Registering TS with master...
I20260925 02:22:59.266745 19855 heartbeater.cc:511] Master 127.11.125.62:42533 requested a full tablet report, sending...
I20260925 02:22:59.267179 19366 ts_manager.cc:194] Registered new tserver with Master: 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:22:59.267759 19366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:55059
I20260925 02:22:59.273207 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.267497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.11.125.3:39443)
0925 02:22:59.267629 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.267640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.267700 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.267843 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.267844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.268048 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:59.268114 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.268116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.268983 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.268984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.269485 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.269486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.269800 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.269801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.269847 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.270286 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:59.270291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:59.270680 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:59.272215 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.272216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.272216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.272395 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.272596 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.272597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.272597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.272669 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.272894 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:59.272894 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:59.273000 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.273056 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.273134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260925 02:22:59.273209 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.267562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:39443 (local address 127.11.125.62:42533)
0925 02:22:59.267608 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.267610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.267692 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:22:59.267750 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.267750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.267761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.267821 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.268206 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.268843 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.269553 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.269729 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.269786 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.270745 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.270746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.270746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.270752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.272095 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.272461 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.272463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.272463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.272527 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.272727 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.272728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.272728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.272837 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.272888 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.273092 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.273151 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:22:59.275295 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:22:59.296810 19416 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:59.291081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.11.125.62:42533)
0925 02:22:59.291132 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.291134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.291146 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:22:59.291211 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.291212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.291223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.291273 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.291834 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.292331 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.293104 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.293289 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.293343 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.296666 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52754: BlockingRecv error: recv got EOF from 127.0.0.1:52754 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:59.304382 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.297650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.11.125.62:42533)
0925 02:22:59.297714 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.297717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.297758 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:22:59.297842 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.297843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.297853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.297924 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.298407 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.298896 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.299584 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.299783 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.299857 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.302041 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.302042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.302042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.302049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.303517 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.303819 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.303820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.303820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.303871 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.304016 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.304016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.304017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.304111 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.304183 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.304259 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.304317 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:22:59.306028 19366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:59.307868 19366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:59.314579 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.11.125.1:39249)
0925 02:22:59.310751 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.310757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.310830 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:22:59.310909 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.310911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.310930 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.311006 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.311228 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.311998 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.313489 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314121 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.314309 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.314438 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.314482 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 02:22:59.314567 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39249 (local address 127.0.0.1:40786)
0925 02:22:59.310780 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.310789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.310842 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.311082 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.311083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.311085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.311147 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.311149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.312096 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.312097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.313392 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.313395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314212 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.314296 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.314313 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.314416 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.314475 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260925 02:22:59.314829 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:34729 (local address 127.0.0.1:50442)
0925 02:22:59.310448 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.310494 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.310562 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.311035 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.311038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.311041 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.311118 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.311120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.312542 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.312543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.313687 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.313690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314602 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.314671 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.314675 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.314703 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.314745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":29,"thread_start_us":103,"threads_started":1}
I20260925 02:22:59.315083 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:43509 (local address 127.0.0.1:33360)
0925 02:22:59.310918 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.310927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.310969 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.311224 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.311225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.311226 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.311256 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.311257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.312550 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.312551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.313652 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.313655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314867 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.314930 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.314934 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.314983 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.315023 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":33,"thread_start_us":109,"threads_started":1}
I20260925 02:22:59.315384 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.11.125.2:43509)
0925 02:22:59.310745 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.310750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.311028 (+ 278us) server_negotiation.cc:407] Connection header received
0925 02:22:59.311073 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.311075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.311089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.311158 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.311351 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.312443 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.313745 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314478 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314555 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.314634 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.315227 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.315282 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260925 02:22:59.315512 19506 tablet_service.cc:1467] Processing CreateTablet for tablet a93d1bf322854195bb7617df37fd2591 (DEFAULT_TABLE table=test-table [id=972c534f4c574f3ca2781e66ebfb2811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.315946 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d1bf322854195bb7617df37fd2591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.316085 19647 tablet_service.cc:1467] Processing CreateTablet for tablet a93d1bf322854195bb7617df37fd2591 (DEFAULT_TABLE table=test-table [id=972c534f4c574f3ca2781e66ebfb2811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.316344 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d1bf322854195bb7617df37fd2591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.317235 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.310471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.11.125.3:34729)
0925 02:22:59.310802 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.310806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.310816 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:22:59.310861 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.310863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.310876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.310976 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.311343 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.312455 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.313808 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.314510 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.314940 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.317054 (+ 2114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.317091 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.317141 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
I20260925 02:22:59.317833 19788 tablet_service.cc:1467] Processing CreateTablet for tablet a93d1bf322854195bb7617df37fd2591 (DEFAULT_TABLE table=test-table [id=972c534f4c574f3ca2781e66ebfb2811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.318181 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d1bf322854195bb7617df37fd2591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.319839 19883 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Bootstrap starting.
I20260925 02:22:59.320384 19883 tablet_bootstrap.cc:654] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.320616 19883 log.cc:824] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:59.321345 19884 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Bootstrap starting.
I20260925 02:22:59.321379 19883 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: No bootstrap required, opened a new log
I20260925 02:22:59.321467 19883 ts_tablet_manager.cc:1397] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:59.321802 19886 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Bootstrap starting.
I20260925 02:22:59.322016 19884 tablet_bootstrap.cc:654] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.322365 19884 log.cc:824] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:59.322513 19886 tablet_bootstrap.cc:654] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.322801 19886 log.cc:824] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:22:59.323268 19883 raft_consensus.cc:348] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.323436 19883 raft_consensus.cc:374] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.323482 19883 raft_consensus.cc:729] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deabff8b4a54f28832359d889f105d0, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.323489 19886 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: No bootstrap required, opened a new log
I20260925 02:22:59.323561 19886 ts_tablet_manager.cc:1397] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:59.323607 19883 consensus_queue.cc:260] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.323902 19883 ts_tablet_manager.cc:1428] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:22:59.323948 19714 heartbeater.cc:503] Master 127.11.125.62:42533 was elected leader, sending a full tablet report...
I20260925 02:22:59.324599 19884 tablet_bootstrap.cc:492] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: No bootstrap required, opened a new log
I20260925 02:22:59.324669 19884 ts_tablet_manager.cc:1397] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:22:59.326117 19886 raft_consensus.cc:348] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.326292 19886 raft_consensus.cc:374] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.326328 19886 raft_consensus.cc:729] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.326452 19886 consensus_queue.cc:260] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.326730 19886 ts_tablet_manager.cc:1428] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:22:59.326783 19855 heartbeater.cc:503] Master 127.11.125.62:42533 was elected leader, sending a full tablet report...
I20260925 02:22:59.326745 19884 raft_consensus.cc:348] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.326902 19884 raft_consensus.cc:374] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.326927 19884 raft_consensus.cc:729] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.327041 19884 consensus_queue.cc:260] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.327310 19884 ts_tablet_manager.cc:1428] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:22:59.327351 19573 heartbeater.cc:503] Master 127.11.125.62:42533 was elected leader, sending a full tablet report...
W20260925 02:22:59.352802 19715 tablet.cc:2369] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:22:59.449270 19574 tablet.cc:2369] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:59.475662 19890 raft_consensus.cc:482] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:59.475891 19890 raft_consensus.cc:504] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.476291 19890 leader_election.cc:288] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249)
I20260925 02:22:59.479614 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.476642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34169 (local address 127.11.125.1:39249)
0925 02:22:59.476687 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.476689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.476823 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:22:59.476946 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.476947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.476957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.477038 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.477213 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.477794 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.478856 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.479225 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.479328 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.479338 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.479495 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.479531 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:22:59.479630 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.476597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:36987 (local address 127.11.125.2:43509)
0925 02:22:59.476647 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.476649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.476664 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:22:59.476763 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.476765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.476784 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.476856 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.477087 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.477648 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.478797 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.479209 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.479332 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.479341 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.479508 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.479550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:22:59.479648 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.476543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:43509 (local address 127.11.125.3:36987)
0925 02:22:59.476606 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.476622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.476679 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.476878 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.476879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.476881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.477006 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.477008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.477747 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.477749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.478698 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.478701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.479340 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.479341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.479427 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.479430 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.479503 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.479547 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:22:59.479674 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.476561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39249 (local address 127.11.125.3:34169)
0925 02:22:59.476742 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.476751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.476841 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.477045 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.477046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.477048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.477133 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.477134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.477872 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.477873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.478711 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.478714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.479328 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.479330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.479417 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.479421 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.479489 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.479531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260925 02:22:59.480053 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d1bf322854195bb7617df37fd2591" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0" is_pre_election: true
I20260925 02:22:59.480051 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d1bf322854195bb7617df37fd2591" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1401078530e47e7a24c684135ffe3ff" is_pre_election: true
I20260925 02:22:59.480239 19667 raft_consensus.cc:2454] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 0.
I20260925 02:22:59.480239 19526 raft_consensus.cc:2454] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 0.
I20260925 02:22:59.480527 19742 leader_election.cc:302] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0a1402f42f47bebfa49104d937e96f, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.480671 19890 raft_consensus.cc:2781] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:59.480731 19890 raft_consensus.cc:482] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:59.480754 19890 raft_consensus.cc:3037] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.481796 19890 raft_consensus.cc:504] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.481949 19890 leader_election.cc:288] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 election: Requested vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249)
I20260925 02:22:59.482153 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d1bf322854195bb7617df37fd2591" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1401078530e47e7a24c684135ffe3ff"
I20260925 02:22:59.482164 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d1bf322854195bb7617df37fd2591" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0"
I20260925 02:22:59.482257 19526 raft_consensus.cc:3037] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.482254 19667 raft_consensus.cc:3037] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.482936 19526 raft_consensus.cc:2454] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 1.
I20260925 02:22:59.483094 19667 raft_consensus.cc:2454] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 1.
I20260925 02:22:59.483171 19742 leader_election.cc:302] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0a1402f42f47bebfa49104d937e96f, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.483261 19890 raft_consensus.cc:2781] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:59.483429 19890 raft_consensus.cc:686] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 1 LEADER]: Becoming Leader. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Running, Role: LEADER
I20260925 02:22:59.483539 19890 consensus_queue.cc:237] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } }
I20260925 02:22:59.484498 19365 catalog_manager.cc:5495] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3). New cstate: current_term: 1 leader_uuid: "1f0a1402f42f47bebfa49104d937e96f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:59.496110 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.493437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.11.125.3:34729)
0925 02:22:59.493499 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.493501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.493526 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:22:59.493614 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.493615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.493635 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.493725 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:59.493889 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.494370 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.495245 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.495458 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.495539 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.495730 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.495879 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.495979 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.495999 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.496019 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:22:59.498181 19667 raft_consensus.cc:1261] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Refusing update from remote peer 1f0a1402f42f47bebfa49104d937e96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.498548 19526 raft_consensus.cc:1261] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Refusing update from remote peer 1f0a1402f42f47bebfa49104d937e96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.498554 19893 consensus_queue.cc:1035] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.498833 19893 consensus_queue.cc:1035] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.501410 19896 mvcc.cc:205] Tried to move back new op lower bound from 7333081004021751808 to 7333081003965415424. Current Snapshot: MvccSnapshot[applied={T|T < 7333081004021751808}]
I20260925 02:22:59.501449 19898 mvcc.cc:205] Tried to move back new op lower bound from 7333081004021751808 to 7333081003965415424. Current Snapshot: MvccSnapshot[applied={T|T < 7333081004021751808}]
I20260925 02:22:59.503621 19366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:59.503718 19366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:59.504544 19366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Sending DeleteTablet for 3 replicas of tablet a93d1bf322854195bb7617df37fd2591
I20260925 02:22:59.505016 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d1bf322854195bb7617df37fd2591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:33360
I20260925 02:22:59.505028 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d1bf322854195bb7617df37fd2591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260925 02:22:59.505300 19906 tablet_replica.cc:330] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: stopping tablet replica
I20260925 02:22:59.505411 19906 raft_consensus.cc:2229] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:59.505455 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d1bf322854195bb7617df37fd2591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260925 02:22:59.505506 19906 raft_consensus.cc:2258] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.505472 19366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:59.505630 19366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:59.505875 19907 tablet_replica.cc:330] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: stopping tablet replica
I20260925 02:22:59.506001 19907 raft_consensus.cc:2229] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:59.506129 19907 raft_consensus.cc:2258] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.506143 19906 ts_tablet_manager.cc:1905] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.506619 19907 ts_tablet_manager.cc:1905] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.507755 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 083b674119a5428698e5cedfa80693fd (DEFAULT_TABLE table=test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.507872 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083b674119a5428698e5cedfa80693fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.508467 19906 ts_tablet_manager.cc:1918] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:59.508476 19907 ts_tablet_manager.cc:1918] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:59.508546 19907 log.cc:1196] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/a93d1bf322854195bb7617df37fd2591
I20260925 02:22:59.508546 19906 log.cc:1196] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/a93d1bf322854195bb7617df37fd2591
I20260925 02:22:59.508812 19907 ts_tablet_manager.cc:1939] T a93d1bf322854195bb7617df37fd2591 P 0deabff8b4a54f28832359d889f105d0: Deleting consensus metadata
I20260925 02:22:59.508875 19906 ts_tablet_manager.cc:1939] T a93d1bf322854195bb7617df37fd2591 P b1401078530e47e7a24c684135ffe3ff: Deleting consensus metadata
I20260925 02:22:59.509254 19350 catalog_manager.cc:4841] TS 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509): tablet a93d1bf322854195bb7617df37fd2591 (table test-table [id=972c534f4c574f3ca2781e66ebfb2811]) successfully deleted
I20260925 02:22:59.509359 19353 catalog_manager.cc:4841] TS b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249): tablet a93d1bf322854195bb7617df37fd2591 (table test-table [id=972c534f4c574f3ca2781e66ebfb2811]) successfully deleted
I20260925 02:22:59.509387 19884 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Bootstrap starting.
I20260925 02:22:59.509421 19647 tablet_service.cc:1467] Processing CreateTablet for tablet 083b674119a5428698e5cedfa80693fd (DEFAULT_TABLE table=test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.509527 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083b674119a5428698e5cedfa80693fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.509779 19884 tablet_bootstrap.cc:654] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.510452 19884 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: No bootstrap required, opened a new log
I20260925 02:22:59.510509 19884 ts_tablet_manager.cc:1397] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:59.510643 19884 raft_consensus.cc:348] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.510716 19884 raft_consensus.cc:374] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.510741 19884 raft_consensus.cc:729] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.510753 19883 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Bootstrap starting.
I20260925 02:22:59.510782 19884 consensus_queue.cc:260] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.510883 19884 ts_tablet_manager.cc:1428] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:59.510967 19788 tablet_service.cc:1467] Processing CreateTablet for tablet 083b674119a5428698e5cedfa80693fd (DEFAULT_TABLE table=test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.511096 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083b674119a5428698e5cedfa80693fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.511224 19883 tablet_bootstrap.cc:654] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.511931 19883 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: No bootstrap required, opened a new log
I20260925 02:22:59.511994 19883 ts_tablet_manager.cc:1397] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:59.512161 19883 raft_consensus.cc:348] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.512243 19883 raft_consensus.cc:374] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.512264 19883 raft_consensus.cc:729] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deabff8b4a54f28832359d889f105d0, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.512318 19883 consensus_queue.cc:260] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.512423 19883 ts_tablet_manager.cc:1428] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:59.512666 19886 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Bootstrap starting.
I20260925 02:22:59.513257 19886 tablet_bootstrap.cc:654] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.514477 19886 tablet_bootstrap.cc:492] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: No bootstrap required, opened a new log
I20260925 02:22:59.514546 19886 ts_tablet_manager.cc:1397] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:59.514717 19886 raft_consensus.cc:348] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.514801 19886 raft_consensus.cc:374] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.514828 19886 raft_consensus.cc:729] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.514889 19886 consensus_queue.cc:260] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.514987 19886 ts_tablet_manager.cc:1428] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:59.515328 19908 tablet_replica.cc:330] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: stopping tablet replica
I20260925 02:22:59.515441 19908 raft_consensus.cc:2229] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:59.515547 19908 raft_consensus.cc:2258] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.515947 19908 ts_tablet_manager.cc:1905] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.517570 19908 ts_tablet_manager.cc:1918] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:22:59.517722 19908 log.cc:1196] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/a93d1bf322854195bb7617df37fd2591
I20260925 02:22:59.518031 19908 ts_tablet_manager.cc:1939] T a93d1bf322854195bb7617df37fd2591 P 1f0a1402f42f47bebfa49104d937e96f: Deleting consensus metadata
I20260925 02:22:59.518486 19351 catalog_manager.cc:4841] TS 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729): tablet a93d1bf322854195bb7617df37fd2591 (table test-table [id=972c534f4c574f3ca2781e66ebfb2811]) successfully deleted
I20260925 02:22:59.666860 19891 raft_consensus.cc:482] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:59.667006 19891 raft_consensus.cc:504] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.667412 19891 leader_election.cc:288] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:22:59.670519 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.667695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:40691 (local address 127.11.125.2:43509)
0925 02:22:59.667751 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.667754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.667845 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:22:59.667938 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.667940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.667950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.668007 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.668195 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.668694 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.669740 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.670114 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.670172 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.670179 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.670407 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.670448 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:22:59.670531 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.667735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42121 (local address 127.11.125.3:34729)
0925 02:22:59.667799 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.667801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.667909 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:22:59.667950 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.667955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.667966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.668038 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:22:59.668233 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.668693 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.669779 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.670127 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.670221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.670230 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.670432 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.670472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:22:59.670534 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.667619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:43509 (local address 127.11.125.1:40691)
0925 02:22:59.667766 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.667778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.667859 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.668023 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.668024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.668026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.668120 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.668122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.668782 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.668783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.669646 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.669649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.670217 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.670218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.670308 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.670345 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.670398 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.670445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260925 02:22:59.670825 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.667619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:34729 (local address 127.11.125.1:42121)
0925 02:22:59.667819 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.667830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.667899 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.668056 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.668057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.668059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:22:59.668134 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.668136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.668805 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.668806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.669651 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.669653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.670232 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.670233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.670362 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.670367 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.670425 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.670498 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260925 02:22:59.670801 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083b674119a5428698e5cedfa80693fd" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0" is_pre_election: true
I20260925 02:22:59.670889 19667 raft_consensus.cc:2454] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:22:59.671118 19456 leader_election.cc:302] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.671159 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083b674119a5428698e5cedfa80693fd" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f" is_pre_election: true
I20260925 02:22:59.671262 19891 raft_consensus.cc:2781] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:59.671321 19891 raft_consensus.cc:482] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:59.671319 19808 raft_consensus.cc:2454] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:22:59.671351 19891 raft_consensus.cc:3037] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.672139 19891 raft_consensus.cc:504] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.672304 19891 leader_election.cc:288] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Requested vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:22:59.672429 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083b674119a5428698e5cedfa80693fd" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0"
I20260925 02:22:59.672456 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083b674119a5428698e5cedfa80693fd" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f"
I20260925 02:22:59.672509 19667 raft_consensus.cc:3037] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.672523 19808 raft_consensus.cc:3037] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.673326 19667 raft_consensus.cc:2454] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:22:59.673326 19808 raft_consensus.cc:2454] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:22:59.673525 19456 leader_election.cc:302] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.673648 19891 raft_consensus.cc:2781] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:59.673820 19891 raft_consensus.cc:686] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 1 LEADER]: Becoming Leader. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Running, Role: LEADER
I20260925 02:22:59.673910 19891 consensus_queue.cc:237] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.674690 19366 catalog_manager.cc:5495] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to b1401078530e47e7a24c684135ffe3ff (127.11.125.1). New cstate: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302978, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.62@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:59.720747 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.713695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42533 (local address 127.0.0.1:52782)
0925 02:22:59.713897 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.713909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.713976 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.714095 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.714096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.714430 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:22:59.714495 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.714506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.715155 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.715156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.715801 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.715804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.716133 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.716134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.716187 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.716763 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:22:59.716768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:22:59.718235 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:22:59.719684 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.719694 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.719697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.719960 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.720155 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:22:59.720156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:22:59.720156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:22:59.720217 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:22:59.720441 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:22:59.720447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:22:59.720491 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.720543 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.720647 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260925 02:22:59.720798 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.713743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.11.125.62:42533)
0925 02:22:59.713832 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.713834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.713977 (+ 143us) server_negotiation.cc:407] Connection header received
0925 02:22:59.714016 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.714017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.714027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.714077 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.714579 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.715084 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.715888 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.716064 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.716116 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.718333 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.718334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.718334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.718340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.719600 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.720038 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.720039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.720040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.720092 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.720276 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.720277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.720277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.720384 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.720435 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.720580 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.720696 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":61}
I20260925 02:22:59.722909 19366 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:52782:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:22:59.723098 19366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:59.724639 19647 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7dc12923f479a8a5ab25c267c5efe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4174338e6e1f45b8bb7eecc9eba5f9c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:59.724639 19506 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7dc12923f479a8a5ab25c267c5efe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4174338e6e1f45b8bb7eecc9eba5f9c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:59.724820 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7dc12923f479a8a5ab25c267c5efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.724825 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7dc12923f479a8a5ab25c267c5efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.725036 19788 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7dc12923f479a8a5ab25c267c5efe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4174338e6e1f45b8bb7eecc9eba5f9c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:22:59.725207 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7dc12923f479a8a5ab25c267c5efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.726390 19886 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: Bootstrap starting.
I20260925 02:22:59.726445 19883 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: Bootstrap starting.
I20260925 02:22:59.726805 19886 tablet_bootstrap.cc:654] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.726904 19883 tablet_bootstrap.cc:654] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.727243 19884 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: Bootstrap starting.
I20260925 02:22:59.727394 19886 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: No bootstrap required, opened a new log
I20260925 02:22:59.727452 19886 ts_tablet_manager.cc:1397] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:59.727568 19883 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: No bootstrap required, opened a new log
I20260925 02:22:59.727581 19886 raft_consensus.cc:348] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.727615 19883 ts_tablet_manager.cc:1397] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:59.727633 19886 raft_consensus.cc:374] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.727648 19886 raft_consensus.cc:729] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.727659 19884 tablet_bootstrap.cc:654] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.727742 19886 consensus_queue.cc:260] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.727774 19883 raft_consensus.cc:348] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.727840 19886 ts_tablet_manager.cc:1428] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.727851 19883 raft_consensus.cc:374] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.727877 19883 raft_consensus.cc:729] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deabff8b4a54f28832359d889f105d0, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.727856 19893 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.727954 19893 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.727946 19883 consensus_queue.cc:260] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.728091 19883 ts_tablet_manager.cc:1428] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.728134 19889 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.728223 19889 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.728839 19884 tablet_bootstrap.cc:492] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: No bootstrap required, opened a new log
I20260925 02:22:59.728909 19884 ts_tablet_manager.cc:1397] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:22:59.729076 19884 raft_consensus.cc:348] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.729146 19884 raft_consensus.cc:374] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.729171 19884 raft_consensus.cc:729] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.729210 19884 consensus_queue.cc:260] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.729277 19884 ts_tablet_manager.cc:1428] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:22:59.729348 19891 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.729452 19891 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 02:22:59.763547 19856 tablet.cc:2369] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:22:59.796682 19891 raft_consensus.cc:482] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:22:59.796790 19891 raft_consensus.cc:504] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.796998 19891 leader_election.cc:288] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:22:59.797263 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7dc12923f479a8a5ab25c267c5efe" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f" is_pre_election: true
I20260925 02:22:59.797248 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7dc12923f479a8a5ab25c267c5efe" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0" is_pre_election: true
I20260925 02:22:59.797410 19808 raft_consensus.cc:2454] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:22:59.797436 19667 raft_consensus.cc:2454] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:22:59.797585 19457 leader_election.cc:302] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0a1402f42f47bebfa49104d937e96f, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.797689 19891 raft_consensus.cc:2781] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:22:59.797744 19891 raft_consensus.cc:482] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:22:59.797757 19891 raft_consensus.cc:3037] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.798638 19891 raft_consensus.cc:504] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.798805 19891 leader_election.cc:288] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Requested vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:22:59.798997 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7dc12923f479a8a5ab25c267c5efe" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0"
I20260925 02:22:59.798990 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7dc12923f479a8a5ab25c267c5efe" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f"
I20260925 02:22:59.799070 19667 raft_consensus.cc:3037] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.799072 19808 raft_consensus.cc:3037] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:22:59.799818 19667 raft_consensus.cc:2454] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:22:59.799818 19808 raft_consensus.cc:2454] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:22:59.799994 19456 leader_election.cc:302] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:22:59.800141 19891 raft_consensus.cc:2781] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:22:59.800221 19891 raft_consensus.cc:686] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [term 1 LEADER]: Becoming Leader. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Running, Role: LEADER
I20260925 02:22:59.800273 19891 consensus_queue.cc:237] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.800596 19914 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: TxnStatusTablet state changed. Reason: New leader b1401078530e47e7a24c684135ffe3ff. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.800711 19914 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:59.800930 19926 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:22:59.800872 19366 catalog_manager.cc:5495] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to b1401078530e47e7a24c684135ffe3ff (127.11.125.1). New cstate: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } health_report { overall_health: UNKNOWN } } }
I20260925 02:22:59.801049 19926 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:22:59.822099 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.819547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.11.125.1:39249)
0925 02:22:59.819590 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.819593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.819601 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:59.819684 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.819685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.819713 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.819800 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:59.819946 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.820432 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.821266 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.821456 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.821531 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.821725 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.821838 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.821925 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.821979 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.822000 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:59.822103 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.819431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39249 (local address 127.0.0.1:40792)
0925 02:22:59.819516 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.819530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.819589 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.819810 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.819811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.819812 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:59.819873 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.819874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.820519 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.820520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.821173 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.821175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.821528 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.821530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.821601 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.821616 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.821908 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.821918 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.821974 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.822028 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:22:59.823418 19808 raft_consensus.cc:1261] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.823436 19667 raft_consensus.cc:1261] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.823716 19914 consensus_queue.cc:1035] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.823830 19891 consensus_queue.cc:1035] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.824345 19889 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: TxnStatusTablet state changed. Reason: New leader b1401078530e47e7a24c684135ffe3ff. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.824438 19889 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.824504 19893 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: TxnStatusTablet state changed. Reason: New leader b1401078530e47e7a24c684135ffe3ff. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.824595 19893 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.824994 19891 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.825084 19891 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:59.825167 19897 mvcc.cc:205] Tried to move back new op lower bound from 7333081005354061824 to 7333081005262188544. Current Snapshot: MvccSnapshot[applied={T|T < 7333081005354061824 or (T in {7333081005354061824})}]
I20260925 02:22:59.825690 19891 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.825760 19891 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P b1401078530e47e7a24c684135ffe3ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:22:59.825914 19893 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.825994 19893 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 1f0a1402f42f47bebfa49104d937e96f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.827512 19889 tablet_replica.cc:439] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } } }
I20260925 02:22:59.827587 19889 tablet_replica.cc:442] T e0c7dc12923f479a8a5ab25c267c5efe P 0deabff8b4a54f28832359d889f105d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:22:59.829387 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.826862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.11.125.1:39249)
0925 02:22:59.826902 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.826904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.826912 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:22:59.826978 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.826979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.826988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.827041 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:59.827205 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.827858 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.828645 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.828985 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.829064 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.829161 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.829194 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.829246 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.829300 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.829319 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:22:59.835006 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.832064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:35481 (local address 127.11.125.1:39249)
0925 02:22:59.832147 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.832149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.832270 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:22:59.832312 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.832312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.832323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.832380 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:22:59.832748 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.833499 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.834244 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.834533 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.834603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.834757 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.834796 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.834848 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.834896 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.834917 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:22:59.835019 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.832020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39249 (local address 127.11.125.1:35481)
0925 02:22:59.832190 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:22:59.832201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:22:59.832266 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:22:59.832391 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:22:59.832392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:22:59.832393 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:22:59.832680 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.832681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.833575 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.833576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.834165 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:22:59.834166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.834624 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.834625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:22:59.834667 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.834681 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:22:59.834857 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:22:59.834858 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 02:22:59.834893 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 02:22:59.834928 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260925 02:22:59.837181 19667 raft_consensus.cc:1261] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.837220 19808 raft_consensus.cc:1261] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:22:59.837373 19914 consensus_queue.cc:1035] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.837450 19891 consensus_queue.cc:1035] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:22:59.851976 19366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:59.852083 19366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:22:59.852690 19366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Sending DeleteTablet for 3 replicas of tablet 083b674119a5428698e5cedfa80693fd
I20260925 02:22:59.853052 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet 083b674119a5428698e5cedfa80693fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260925 02:22:59.853163 19908 tablet_replica.cc:330] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: stopping tablet replica
I20260925 02:22:59.853236 19908 raft_consensus.cc:2229] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:59.853255 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet 083b674119a5428698e5cedfa80693fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:33360
I20260925 02:22:59.853296 19908 raft_consensus.cc:2258] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.853338 19907 tablet_replica.cc:330] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: stopping tablet replica
I20260925 02:22:59.853380 19907 raft_consensus.cc:2229] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:22:59.853415 19907 raft_consensus.cc:2258] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.853857 19907 ts_tablet_manager.cc:1905] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.855880 19907 ts_tablet_manager.cc:1918] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:59.855940 19907 log.cc:1196] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/083b674119a5428698e5cedfa80693fd
I20260925 02:22:59.856169 19907 ts_tablet_manager.cc:1939] T 083b674119a5428698e5cedfa80693fd P 0deabff8b4a54f28832359d889f105d0: Deleting consensus metadata
I20260925 02:22:59.856303 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 083b674119a5428698e5cedfa80693fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:22:59 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260925 02:22:59.856365 19908 ts_tablet_manager.cc:1905] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.856396 19906 tablet_replica.cc:330] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: stopping tablet replica
I20260925 02:22:59.856434 19906 raft_consensus.cc:2229] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:22:59.856474 19350 catalog_manager.cc:4841] TS 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509): tablet 083b674119a5428698e5cedfa80693fd (table test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]) successfully deleted
I20260925 02:22:59.856547 19906 raft_consensus.cc:2258] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:22:59.856865 19906 ts_tablet_manager.cc:1905] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:22:59.857981 19908 ts_tablet_manager.cc:1918] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:59.858047 19908 log.cc:1196] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/083b674119a5428698e5cedfa80693fd
I20260925 02:22:59.858245 19908 ts_tablet_manager.cc:1939] T 083b674119a5428698e5cedfa80693fd P 1f0a1402f42f47bebfa49104d937e96f: Deleting consensus metadata
I20260925 02:22:59.858577 19906 ts_tablet_manager.cc:1918] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:22:59.858636 19906 log.cc:1196] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/083b674119a5428698e5cedfa80693fd
I20260925 02:22:59.858834 19906 ts_tablet_manager.cc:1939] T 083b674119a5428698e5cedfa80693fd P b1401078530e47e7a24c684135ffe3ff: Deleting consensus metadata
I20260925 02:22:59.860040 19351 catalog_manager.cc:4841] TS 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729): tablet 083b674119a5428698e5cedfa80693fd (table test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]) successfully deleted
I20260925 02:22:59.860394 19353 catalog_manager.cc:4841] TS b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249): tablet 083b674119a5428698e5cedfa80693fd (table test-table [id=1bf65f6b04384a6888c0de0bb42ff47f]) successfully deleted
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:22:59.889678 19416 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:22:59.882429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.11.125.62:42533)
0925 02:22:59.882481 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.882484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.882620 (+ 136us) server_negotiation.cc:407] Connection header received
0925 02:22:59.882705 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.882706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.882720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.882806 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.883565 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.884427 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.885701 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.886011 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.886098 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.889601 (+ 3503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52790: BlockingRecv error: recv got EOF from 127.0.0.1:52790 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 02:22:59 dist-test-slave-hj7s krb5kdc[19321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM
I20260925 02:22:59.900682 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:22:59.891127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.11.125.62:42533)
0925 02:22:59.891167 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:22:59.891169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:22:59.891437 (+ 268us) server_negotiation.cc:407] Connection header received
0925 02:22:59.891547 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:22:59.891548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:22:59.891559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:22:59.891619 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:22:59.892208 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.893049 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.894016 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:22:59.894319 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:22:59.894394 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:22:59.897368 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:22:59.897370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:22:59.897370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:22:59.897378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:22:59.899234 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.899666 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.899667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.899667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.899734 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:22:59.899958 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:22:59.899958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:22:59.899959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:22:59.900108 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:22:59.900192 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:22:59.900456 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 02:22:59.900546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:22:59.902829 19366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:22:59.903019 19366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:22:59.904548 19788 tablet_service.cc:1467] Processing CreateTablet for tablet 46a457f77e4b4e1da466cab63c07e829 (DEFAULT_TABLE table=test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.904548 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 46a457f77e4b4e1da466cab63c07e829 (DEFAULT_TABLE table=test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.904685 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a457f77e4b4e1da466cab63c07e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.904685 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a457f77e4b4e1da466cab63c07e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.904702 19647 tablet_service.cc:1467] Processing CreateTablet for tablet 46a457f77e4b4e1da466cab63c07e829 (DEFAULT_TABLE table=test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:22:59.904872 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a457f77e4b4e1da466cab63c07e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:22:59.906045 19884 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Bootstrap starting.
I20260925 02:22:59.906068 19886 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Bootstrap starting.
I20260925 02:22:59.906339 19883 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Bootstrap starting.
I20260925 02:22:59.906430 19886 tablet_bootstrap.cc:654] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.906447 19884 tablet_bootstrap.cc:654] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.906769 19883 tablet_bootstrap.cc:654] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Neither blocks nor log segments found. Creating new log.
I20260925 02:22:59.907111 19886 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: No bootstrap required, opened a new log
I20260925 02:22:59.907141 19884 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: No bootstrap required, opened a new log
I20260925 02:22:59.907169 19886 ts_tablet_manager.cc:1397] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:22:59.907217 19884 ts_tablet_manager.cc:1397] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:22:59.907310 19886 raft_consensus.cc:348] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.907390 19886 raft_consensus.cc:374] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.907419 19886 raft_consensus.cc:729] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.907393 19884 raft_consensus.cc:348] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.907467 19884 raft_consensus.cc:374] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.907460 19886 consensus_queue.cc:260] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.907495 19884 raft_consensus.cc:729] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.907562 19886 ts_tablet_manager.cc:1428] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:22:59.907543 19884 consensus_queue.cc:260] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.907668 19884 ts_tablet_manager.cc:1428] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 02:22:59.907845 19883 tablet_bootstrap.cc:492] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: No bootstrap required, opened a new log
I20260925 02:22:59.907905 19883 ts_tablet_manager.cc:1397] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:22:59.908074 19883 raft_consensus.cc:348] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.908162 19883 raft_consensus.cc:374] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:22:59.908190 19883 raft_consensus.cc:729] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deabff8b4a54f28832359d889f105d0, State: Initialized, Role: FOLLOWER
I20260925 02:22:59.908236 19883 consensus_queue.cc:260] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:22:59.908352 19883 ts_tablet_manager.cc:1428] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:00.013823 19893 raft_consensus.cc:482] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:23:00.013954 19893 raft_consensus.cc:504] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.014181 19893 leader_election.cc:288] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249), 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509)
I20260925 02:23:00.014442 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a457f77e4b4e1da466cab63c07e829" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0" is_pre_election: true
I20260925 02:23:00.014457 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a457f77e4b4e1da466cab63c07e829" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1401078530e47e7a24c684135ffe3ff" is_pre_election: true
I20260925 02:23:00.014572 19667 raft_consensus.cc:2454] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 0.
I20260925 02:23:00.014575 19526 raft_consensus.cc:2454] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 0.
I20260925 02:23:00.014752 19739 leader_election.cc:302] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, 1f0a1402f42f47bebfa49104d937e96f; no voters:
I20260925 02:23:00.014858 19893 raft_consensus.cc:2781] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:23:00.014917 19893 raft_consensus.cc:482] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:23:00.014940 19893 raft_consensus.cc:3037] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.015908 19893 raft_consensus.cc:504] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.016072 19893 leader_election.cc:288] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 election: Requested vote from peers b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249), 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509)
I20260925 02:23:00.016227 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a457f77e4b4e1da466cab63c07e829" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0"
I20260925 02:23:00.016319 19667 raft_consensus.cc:3037] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.016320 19526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a457f77e4b4e1da466cab63c07e829" candidate_uuid: "1f0a1402f42f47bebfa49104d937e96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1401078530e47e7a24c684135ffe3ff"
I20260925 02:23:00.016405 19526 raft_consensus.cc:3037] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.017217 19667 raft_consensus.cc:2454] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 1.
I20260925 02:23:00.017241 19526 raft_consensus.cc:2454] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0a1402f42f47bebfa49104d937e96f in term 1.
I20260925 02:23:00.017401 19742 leader_election.cc:302] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0a1402f42f47bebfa49104d937e96f, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:23:00.017496 19893 raft_consensus.cc:2781] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:00.017596 19893 raft_consensus.cc:686] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 1 LEADER]: Becoming Leader. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Running, Role: LEADER
I20260925 02:23:00.017652 19893 consensus_queue.cc:237] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.018405 19366 catalog_manager.cc:5495] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3). New cstate: current_term: 1 leader_uuid: "1f0a1402f42f47bebfa49104d937e96f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } health_report { overall_health: HEALTHY } } }
I20260925 02:23:00.027990 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:00.025592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.11.125.3:34729)
0925 02:23:00.025636 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:00.025638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:00.025689 (+ 51us) server_negotiation.cc:407] Connection header received
0925 02:23:00.025767 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:00.025768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:00.025778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:00.025825 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:23:00.025976 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:00.026436 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:00.027252 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:00.027462 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:00.027539 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:00.027718 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:23:00.027759 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:23:00.027820 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:00.027882 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:23:00.027905 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:23:00.028837 19667 raft_consensus.cc:1261] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Refusing update from remote peer 1f0a1402f42f47bebfa49104d937e96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:00.029033 19526 raft_consensus.cc:1261] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Refusing update from remote peer 1f0a1402f42f47bebfa49104d937e96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:00.029135 19893 consensus_queue.cc:1035] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:00.029265 19953 consensus_queue.cc:1035] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:00.031930 19366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:00.032025 19366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:00.032848 19366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Sending DeleteTablet for 3 replicas of tablet 46a457f77e4b4e1da466cab63c07e829
I20260925 02:23:00.033152 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a457f77e4b4e1da466cab63c07e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260925 02:23:00.033152 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a457f77e4b4e1da466cab63c07e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260925 02:23:00.033203 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet 46a457f77e4b4e1da466cab63c07e829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:33360
I20260925 02:23:00.033245 19906 tablet_replica.cc:330] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: stopping tablet replica
I20260925 02:23:00.033245 19908 tablet_replica.cc:330] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: stopping tablet replica
I20260925 02:23:00.033294 19906 raft_consensus.cc:2229] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:00.033293 19907 tablet_replica.cc:330] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: stopping tablet replica
I20260925 02:23:00.033298 19908 raft_consensus.cc:2229] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:23:00.033339 19906 raft_consensus.cc:2258] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.033345 19907 raft_consensus.cc:2229] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:00.033352 19908 raft_consensus.cc:2258] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.033401 19907 raft_consensus.cc:2258] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.033627 19908 ts_tablet_manager.cc:1905] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.034672 19366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:23:00.034853 19366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:23:00.035135 19908 ts_tablet_manager.cc:1918] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:23:00.035210 19908 log.cc:1196] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/46a457f77e4b4e1da466cab63c07e829
I20260925 02:23:00.035362 19906 ts_tablet_manager.cc:1905] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.035475 19908 ts_tablet_manager.cc:1939] T 46a457f77e4b4e1da466cab63c07e829 P 1f0a1402f42f47bebfa49104d937e96f: Deleting consensus metadata
I20260925 02:23:00.035818 19351 catalog_manager.cc:4841] TS 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729): tablet 46a457f77e4b4e1da466cab63c07e829 (table test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]) successfully deleted
I20260925 02:23:00.036479 19506 tablet_service.cc:1467] Processing CreateTablet for tablet d571cf996c1a48f4829b89c9b739727b (DEFAULT_TABLE table=test-table [id=5fd7fa26f2fc495981d76b94bbbf9250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:00.036525 19647 tablet_service.cc:1467] Processing CreateTablet for tablet d571cf996c1a48f4829b89c9b739727b (DEFAULT_TABLE table=test-table [id=5fd7fa26f2fc495981d76b94bbbf9250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:00.036597 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571cf996c1a48f4829b89c9b739727b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:00.036617 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571cf996c1a48f4829b89c9b739727b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:00.037029 19906 ts_tablet_manager.cc:1918] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:23:00.037081 19906 log.cc:1196] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/46a457f77e4b4e1da466cab63c07e829
I20260925 02:23:00.037294 19906 ts_tablet_manager.cc:1939] T 46a457f77e4b4e1da466cab63c07e829 P b1401078530e47e7a24c684135ffe3ff: Deleting consensus metadata
I20260925 02:23:00.037885 19884 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: Bootstrap starting.
I20260925 02:23:00.037967 19353 catalog_manager.cc:4841] TS b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249): tablet 46a457f77e4b4e1da466cab63c07e829 (table test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]) successfully deleted
I20260925 02:23:00.038106 19883 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Bootstrap starting.
I20260925 02:23:00.038183 19884 tablet_bootstrap.cc:654] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:00.038507 19883 tablet_bootstrap.cc:654] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:00.038777 19884 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: No bootstrap required, opened a new log
I20260925 02:23:00.038834 19884 ts_tablet_manager.cc:1397] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:00.038944 19884 raft_consensus.cc:348] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.039007 19884 raft_consensus.cc:374] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:00.039028 19884 raft_consensus.cc:729] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Initialized, Role: FOLLOWER
I20260925 02:23:00.039076 19884 consensus_queue.cc:260] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.039170 19884 ts_tablet_manager.cc:1428] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:00.039569 19788 tablet_service.cc:1467] Processing CreateTablet for tablet d571cf996c1a48f4829b89c9b739727b (DEFAULT_TABLE table=test-table [id=5fd7fa26f2fc495981d76b94bbbf9250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:00.039703 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d571cf996c1a48f4829b89c9b739727b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:00.040620 19886 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Bootstrap starting.
I20260925 02:23:00.041006 19886 tablet_bootstrap.cc:654] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:00.041134 19883 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: No bootstrap required, opened a new log
I20260925 02:23:00.041177 19883 ts_tablet_manager.cc:1397] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:23:00.041332 19883 raft_consensus.cc:348] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.041425 19883 raft_consensus.cc:374] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:00.041447 19883 raft_consensus.cc:729] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0deabff8b4a54f28832359d889f105d0, State: Initialized, Role: FOLLOWER
I20260925 02:23:00.041498 19883 consensus_queue.cc:260] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.041580 19886 tablet_bootstrap.cc:492] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: No bootstrap required, opened a new log
I20260925 02:23:00.041597 19883 ts_tablet_manager.cc:1428] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:00.041620 19886 ts_tablet_manager.cc:1397] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:00.041726 19886 raft_consensus.cc:348] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.041791 19886 raft_consensus.cc:374] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:00.041816 19886 raft_consensus.cc:729] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a1402f42f47bebfa49104d937e96f, State: Initialized, Role: FOLLOWER
I20260925 02:23:00.041869 19886 consensus_queue.cc:260] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.041965 19886 ts_tablet_manager.cc:1428] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:00.042548 19907 ts_tablet_manager.cc:1905] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.043970 19907 ts_tablet_manager.cc:1918] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:23:00.044045 19907 log.cc:1196] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/46a457f77e4b4e1da466cab63c07e829
I20260925 02:23:00.044271 19907 ts_tablet_manager.cc:1939] T 46a457f77e4b4e1da466cab63c07e829 P 0deabff8b4a54f28832359d889f105d0: Deleting consensus metadata
I20260925 02:23:00.044611 19350 catalog_manager.cc:4841] TS 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509): tablet 46a457f77e4b4e1da466cab63c07e829 (table test-table [id=1c03ae5cfe5c4fc4b3cf7a22fc36973c]) successfully deleted
I20260925 02:23:00.103729 19936 raft_consensus.cc:482] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:23:00.103825 19936 raft_consensus.cc:504] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.104015 19936 leader_election.cc:288] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:23:00.104247 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571cf996c1a48f4829b89c9b739727b" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0" is_pre_election: true
I20260925 02:23:00.104245 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571cf996c1a48f4829b89c9b739727b" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f" is_pre_election: true
I20260925 02:23:00.104346 19667 raft_consensus.cc:2454] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:23:00.104355 19808 raft_consensus.cc:2454] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 0.
I20260925 02:23:00.104530 19456 leader_election.cc:302] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:23:00.104656 19936 raft_consensus.cc:2781] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:23:00.104700 19936 raft_consensus.cc:482] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:23:00.104722 19936 raft_consensus.cc:3037] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.105522 19936 raft_consensus.cc:504] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.105700 19936 leader_election.cc:288] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Requested vote from peers 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509), 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729)
I20260925 02:23:00.105832 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571cf996c1a48f4829b89c9b739727b" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0deabff8b4a54f28832359d889f105d0"
I20260925 02:23:00.105931 19667 raft_consensus.cc:3037] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.105935 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d571cf996c1a48f4829b89c9b739727b" candidate_uuid: "b1401078530e47e7a24c684135ffe3ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0a1402f42f47bebfa49104d937e96f"
I20260925 02:23:00.106007 19808 raft_consensus.cc:3037] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:00.106699 19667 raft_consensus.cc:2454] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:23:00.106798 19808 raft_consensus.cc:2454] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1401078530e47e7a24c684135ffe3ff in term 1.
I20260925 02:23:00.106913 19456 leader_election.cc:302] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0deabff8b4a54f28832359d889f105d0, b1401078530e47e7a24c684135ffe3ff; no voters:
I20260925 02:23:00.107010 19936 raft_consensus.cc:2781] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:00.107087 19936 raft_consensus.cc:686] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 1 LEADER]: Becoming Leader. State: Replica: b1401078530e47e7a24c684135ffe3ff, State: Running, Role: LEADER
I20260925 02:23:00.107127 19936 consensus_queue.cc:237] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } }
I20260925 02:23:00.107833 19364 catalog_manager.cc:5495] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff reported cstate change: term changed from 0 to 1, leader changed from <none> to b1401078530e47e7a24c684135ffe3ff (127.11.125.1). New cstate: current_term: 1 leader_uuid: "b1401078530e47e7a24c684135ffe3ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1401078530e47e7a24c684135ffe3ff" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 } health_report { overall_health: UNKNOWN } } }
I20260925 02:23:00.119719 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:00.117579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.11.125.1:39249)
0925 02:23:00.117618 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:00.117620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:00.117629 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:00.117696 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:00.117697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:00.117705 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:00.117743 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:23:00.117845 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:00.118298 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:00.119031 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:00.119235 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:00.119309 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:00.119463 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:23:00.119502 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:23:00.119555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:00.119615 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 02:23:00.119635 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:23:00.122195 19667 raft_consensus.cc:1261] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:00.122185 19808 raft_consensus.cc:1261] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Refusing update from remote peer b1401078530e47e7a24c684135ffe3ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:00.122416 19936 consensus_queue.cc:1035] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0deabff8b4a54f28832359d889f105d0" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:00.122570 19914 consensus_queue.cc:1035] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0a1402f42f47bebfa49104d937e96f" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:00.133133 19364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:00.133257 19364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:00.134008 19364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c64a62148c040d58d3ba6dd4de58242: Sending DeleteTablet for 3 replicas of tablet d571cf996c1a48f4829b89c9b739727b
I20260925 02:23:00.134333 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet d571cf996c1a48f4829b89c9b739727b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:50442
I20260925 02:23:00.134347 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet d571cf996c1a48f4829b89c9b739727b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260925 02:23:00.134382 19647 tablet_service.cc:1514] Processing DeleteTablet for tablet d571cf996c1a48f4829b89c9b739727b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:00 UTC) from {username='oryx', principal='oryx/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:33360
I20260925 02:23:00.134482 19907 tablet_replica.cc:330] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: stopping tablet replica
I20260925 02:23:00.134482 19908 tablet_replica.cc:330] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: stopping tablet replica
I20260925 02:23:00.134552 19907 raft_consensus.cc:2229] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:00.134552 19908 raft_consensus.cc:2229] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:00.134601 19907 raft_consensus.cc:2258] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.134600 19908 raft_consensus.cc:2258] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.134702 19906 tablet_replica.cc:330] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: stopping tablet replica
I20260925 02:23:00.134768 19906 raft_consensus.cc:2229] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:23:00.134840 19906 raft_consensus.cc:2258] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:00.134920 19907 ts_tablet_manager.cc:1905] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.135151 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 19440
I20260925 02:23:00.135171 19906 ts_tablet_manager.cc:1905] T d571cf996c1a48f4829b89c9b739727b P b1401078530e47e7a24c684135ffe3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.135191 19908 ts_tablet_manager.cc:1905] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:00.136714 19908 ts_tablet_manager.cc:1918] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:23:00.136761 19908 log.cc:1196] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/d571cf996c1a48f4829b89c9b739727b
I20260925 02:23:00.136798 19907 ts_tablet_manager.cc:1918] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:23:00.136853 19907 log.cc:1196] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/d571cf996c1a48f4829b89c9b739727b
I20260925 02:23:00.137002 19908 ts_tablet_manager.cc:1939] T d571cf996c1a48f4829b89c9b739727b P 1f0a1402f42f47bebfa49104d937e96f: Deleting consensus metadata
I20260925 02:23:00.137092 19907 ts_tablet_manager.cc:1939] T d571cf996c1a48f4829b89c9b739727b P 0deabff8b4a54f28832359d889f105d0: Deleting consensus metadata
I20260925 02:23:00.137326 19351 catalog_manager.cc:4841] TS 1f0a1402f42f47bebfa49104d937e96f (127.11.125.3:34729): tablet d571cf996c1a48f4829b89c9b739727b (table test-table [id=5fd7fa26f2fc495981d76b94bbbf9250]) successfully deleted
I20260925 02:23:00.137413 19350 catalog_manager.cc:4841] TS 0deabff8b4a54f28832359d889f105d0 (127.11.125.2:43509): tablet d571cf996c1a48f4829b89c9b739727b (table test-table [id=5fd7fa26f2fc495981d76b94bbbf9250]) successfully deleted
W20260925 02:23:00.144174 19353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:39249 (error 108)
I20260925 02:23:00.144476 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 19582
I20260925 02:23:00.145363 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:00.144728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39249 (local address 127.0.0.1:40814)
0925 02:23:00.145256 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:00.145310 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:39249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":59,"threads_started":1}
W20260925 02:23:00.145925 19353 catalog_manager.cc:4584] TS b1401078530e47e7a24c684135ffe3ff (127.11.125.1:39249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d571cf996c1a48f4829b89c9b739727b: Network error: Client connection negotiation failed: client connection to 127.11.125.1:39249: connect: Connection refused (error 111)
I20260925 02:23:00.150103 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 19723
I20260925 02:23:00.155941 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 19334
2026-09-25T02:23:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2690 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:23:00 dist-test-slave-hj7s krb5kdc[19969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:23:00 dist-test-slave-hj7s krb5kdc[19969](info): set up 2 sockets
Sep 25 02:23:00 dist-test-slave-hj7s krb5kdc[19969](info): commencing operation
krb5kdc: starting...
W20260925 02:23:01.203996 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:23:01 dist-test-slave-hj7s krb5kdc[19969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:23:01.332850 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43829
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:01.417284 19982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:01.417590 19982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:01.417618 19982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:01.418761 19982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:01.418797 19982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:01.418812 19982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:01.418824 19982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:01.418836 19982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:01.420274 19982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:01.420470 19982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:01.420655 19982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:01.425767 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:01.426007 19982 server_base.cc:1034] running on GCE node
W20260925 02:23:01.425961 19988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:01.425783 19987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:01.426371 19982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:01.426723 19982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:01.427947 19982 hybrid_clock.cc:648] HybridClock initialized: now 1790302981427920 us; error 66 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:23:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1274 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:23:01 dist-test-slave-hj7s krb5kdc[19996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:23:01 dist-test-slave-hj7s krb5kdc[19996](info): set up 2 sockets
Sep 25 02:23:01 dist-test-slave-hj7s krb5kdc[19996](info): commencing operation
krb5kdc: starting...
W20260925 02:23:02.488489 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:23:02 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:23:02.616580 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36865
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:02.700703 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:02.701030 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:02.701072 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:02.702446 20009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:02.702510 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:02.702538 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:02.702560 20009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:02.702641 20009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:02.704445 20009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:02.704667 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:02.704855 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:02.710031 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:02.710014 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:02.710088 20009 server_base.cc:1034] running on GCE node
W20260925 02:23:02.710017 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:02.710438 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:02.710772 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:02.711951 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790302982711938 us; error 47 us; skew 500 ppm
Sep 25 02:23:02 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:02.714536 20009 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:23:02.715073 20009 webserver.cc:460] Webserver started at http://127.11.125.62:38209/ using document root <none> and password file <none>
I20260925 02:23:02.715318 20009 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:02.715392 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:02.715552 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:02.716697 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "aa2fff4ff196435aabf32058e1be7ddd"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "133ff6878f9fc74c868056ceee2526e7"
server_key_iv: "d31f2802fdbf1c8af74d4c205a982bc1"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.717146 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "aa2fff4ff196435aabf32058e1be7ddd"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "133ff6878f9fc74c868056ceee2526e7"
server_key_iv: "d31f2802fdbf1c8af74d4c205a982bc1"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.718566 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:02.719390 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:02.719578 20009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:02.719692 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "aa2fff4ff196435aabf32058e1be7ddd"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "133ff6878f9fc74c868056ceee2526e7"
server_key_iv: "d31f2802fdbf1c8af74d4c205a982bc1"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.719800 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:02.741766 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:02.743818 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:02.744007 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:02.748533 20009 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:36865
I20260925 02:23:02.748734 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:36865 every 8 connection(s)
I20260925 02:23:02.749039 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:23:02.749992 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:02.751889 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20009
I20260925 02:23:02.752022 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:23:02.752223 11764 external_mini_cluster.cc:1428] Setting key 3915dcada5b5ed66acaa7ce4c40f0ccd
I20260925 02:23:02.752673 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Bootstrap starting.
I20260925 02:23:02.753427 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:02.753757 20077 log.cc:824] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:02.754462 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: No bootstrap required, opened a new log
I20260925 02:23:02.756302 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:02.756474 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:02.756525 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2fff4ff196435aabf32058e1be7ddd, State: Initialized, Role: FOLLOWER
I20260925 02:23:02.756665 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:02.756752 20077 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:23:02.756796 20077 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:23:02.756845 20077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:02.757426 20077 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:02.757551 20077 leader_election.cc:302] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2fff4ff196435aabf32058e1be7ddd; no voters:
I20260925 02:23:02.757738 20077 leader_election.cc:288] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:23:02.757803 20082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:02.757982 20082 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 LEADER]: Becoming Leader. State: Replica: aa2fff4ff196435aabf32058e1be7ddd, State: Running, Role: LEADER
I20260925 02:23:02.758034 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:23:02.758124 20082 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:02.758513 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2fff4ff196435aabf32058e1be7ddd. Latest consensus state: current_term: 1 leader_uuid: "aa2fff4ff196435aabf32058e1be7ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } } }
I20260925 02:23:02.758752 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:02.758682 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa2fff4ff196435aabf32058e1be7ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } } }
I20260925 02:23:02.759157 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:02.759368 20091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 02:23:02 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:02.759882 20091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:23:02.761739 20091 catalog_manager.cc:1347] Generated new cluster ID: d746c0a8e28848d886130b0af77f189d
I20260925 02:23:02.761812 20091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:23:02.764344 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:02.752833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.11.125.62:36865)
0925 02:23:02.753079 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:02.753089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:02.753129 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:23:02.753686 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:02.753694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:02.753747 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:02.753923 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:02.754685 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.755570 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.756392 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.757081 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.757193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:02.760341 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:02.760345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:02.760346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:02.760358 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:02.762613 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:02.763031 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:02.763035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:02.763035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:02.763147 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:02.763374 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:02.763375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:02.763375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:02.763713 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:02.763883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:02.763970 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 02:23:02.764118 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
I20260925 02:23:02.772030 20091 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:23:02.772547 20091 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:23:02.780038 20091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Generated new TSK 0
I20260925 02:23:02.780234 20091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:23:02.810104 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:02.893318 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:02.893555 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:02.893599 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:02.894804 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:02.894876 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:02.894923 20103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:23:02.896425 20103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:02.896700 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:02.897008 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:02.902143 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:02.902136 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:02.902281 20103 server_base.cc:1034] running on GCE node
W20260925 02:23:02.902137 20111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:02.902639 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:02.902916 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:02.904116 20103 hybrid_clock.cc:648] HybridClock initialized: now 1790302982904102 us; error 28 us; skew 500 ppm
Sep 25 02:23:02 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:02.906908 20103 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:23:02.907407 20103 webserver.cc:460] Webserver started at http://127.11.125.1:37491/ using document root <none> and password file <none>
I20260925 02:23:02.907644 20103 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:02.907689 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:02.907835 20103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:02.908975 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "9f71df7885764fb688cbdf45ce637631"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "9796c6f01ec2469f183ed36d0b3ac344"
server_key_iv: "9dfd525407c7fa9bb219b8cd040e3c65"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.909354 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "9f71df7885764fb688cbdf45ce637631"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "9796c6f01ec2469f183ed36d0b3ac344"
server_key_iv: "9dfd525407c7fa9bb219b8cd040e3c65"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.910745 20103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:23:02.911573 20118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:02.911769 20103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:02.911887 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "9f71df7885764fb688cbdf45ce637631"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "9796c6f01ec2469f183ed36d0b3ac344"
server_key_iv: "9dfd525407c7fa9bb219b8cd040e3c65"
server_key_version: "encryptionkey@0"
I20260925 02:23:02.911963 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:02.922616 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:02.925174 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:02.925338 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:02.925614 20103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:02.925976 20103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:02.926035 20103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:02.926072 20103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:02.926100 20103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:02.932217 20103 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:36719
I20260925 02:23:02.932305 20231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:36719 every 8 connection(s)
I20260925 02:23:02.932622 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:23:02.935377 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20103
I20260925 02:23:02.935539 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:23:02.935750 11764 external_mini_cluster.cc:1428] Setting key bdbcecda34e86cb53214f9472110e96e
Sep 25 02:23:02 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302982, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:02.942834 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:02.933427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:56489 (local address 127.11.125.62:36865)
0925 02:23:02.933493 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:02.933495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:02.933737 (+ 242us) server_negotiation.cc:407] Connection header received
0925 02:23:02.934297 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:02.934298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:02.934312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:02.934376 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:02.935457 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.936108 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.937256 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.937464 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.937578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:02.939876 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:02.939877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:02.939878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:02.939886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:02.941358 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:02.941807 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:02.941808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:02.941809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:02.941875 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:02.942139 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:02.942140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:02.942140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:02.942269 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:02.942344 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:02.942627 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:23:02.942732 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:23:02.943085 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:02.933384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:56489)
0925 02:23:02.933598 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:02.933644 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:02.934178 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:02.934465 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:02.934469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:02.935081 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:02.935341 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:02.935345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.936215 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.936216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:02.937156 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:02.937159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:02.937598 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:02.937599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:02.937736 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:02.938309 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:02.938318 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:02.939731 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:02.941449 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:02.941457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:02.941460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:02.941696 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:02.941949 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:02.941950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:02.941950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:02.942062 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:02.942348 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:02.942355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:02.942493 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:23:02.942607 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:23:02.942812 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260925 02:23:02.943502 20232 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36865
I20260925 02:23:02.943594 20232 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:02.943804 20232 heartbeater.cc:511] Master 127.11.125.62:36865 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:23:02.944415 20041 ts_manager.cc:194] Registered new tserver with Master: 9f71df7885764fb688cbdf45ce637631 (127.11.125.1:36719)
I20260925 02:23:02.945132 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:56489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:23:02.964602 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:03.048018 20237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.048286 20237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.048328 20237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.050050 20237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.050132 20237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.050179 20237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:23:03.051708 20237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.051992 20237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.052227 20237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.057327 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.057338 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.057435 20237 server_base.cc:1034] running on GCE node
W20260925 02:23:03.057318 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.057837 20237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.058111 20237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.059285 20237 hybrid_clock.cc:648] HybridClock initialized: now 1790302983059272 us; error 25 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.061842 20237 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:23:03.062278 20237 webserver.cc:460] Webserver started at http://127.11.125.2:39451/ using document root <none> and password file <none>
I20260925 02:23:03.062522 20237 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.062592 20237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.062742 20237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:03.063796 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "cca3c739882542fcbc5362f737b09ccf"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "675770a2aa35e0efddcbd02530a37f3d"
server_key_iv: "35330431ed0acd7ef97af67c6829fe17"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.064164 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "cca3c739882542fcbc5362f737b09ccf"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "675770a2aa35e0efddcbd02530a37f3d"
server_key_iv: "35330431ed0acd7ef97af67c6829fe17"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.065654 20237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:23:03.066416 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.066589 20237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:03.066696 20237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "cca3c739882542fcbc5362f737b09ccf"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "675770a2aa35e0efddcbd02530a37f3d"
server_key_iv: "35330431ed0acd7ef97af67c6829fe17"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.066774 20237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.087721 20237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.090142 20237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.090323 20237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.090588 20237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:03.090915 20237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:03.090971 20237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.091015 20237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:03.091038 20237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.097131 20237 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36255
I20260925 02:23:03.097266 20367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36255 every 8 connection(s)
I20260925 02:23:03.097558 20237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:23:03.100165 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20237
I20260925 02:23:03.100275 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:23:03.100479 11764 external_mini_cluster.cc:1428] Setting key 4d7d5a88801fcac5f7e1fa0f1a895517
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:03.108229 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.098359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:33211 (local address 127.11.125.62:36865)
0925 02:23:03.098472 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.098476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.098622 (+ 146us) server_negotiation.cc:407] Connection header received
0925 02:23:03.099093 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.099094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.099104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.099162 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.100177 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.100972 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.102085 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.102450 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.102531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.105166 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:03.105168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:03.105168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:03.105176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:03.106755 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.107198 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.107199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.107199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.107262 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.107541 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.107542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.107543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.107664 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:03.107744 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:03.108038 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:23:03.108135 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:23:03.108446 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.098318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:33211)
0925 02:23:03.098482 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.098525 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.099004 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.099257 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.099261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.099874 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.100071 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.100076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.101073 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.101074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.101983 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.101985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.102555 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.102557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.102725 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.103261 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.103273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.105060 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:03.106843 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:03.106853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:03.106855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:03.107107 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:03.107342 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:03.107343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:03.107343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:03.107466 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:03.107735 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:03.107742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:03.107895 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 02:23:03.108014 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:23:03.108182 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260925 02:23:03.109045 20368 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36865
I20260925 02:23:03.109166 20368 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:23:03.109377 20368 heartbeater.cc:511] Master 127.11.125.62:36865 requested a full tablet report, sending...
I20260925 02:23:03.109836 20041 ts_manager.cc:194] Registered new tserver with Master: cca3c739882542fcbc5362f737b09ccf (127.11.125.2:36255)
I20260925 02:23:03.110229 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:33211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:23:03.128969 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:03.213052 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.213344 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.213373 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.214690 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.214745 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.214773 20373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:23:03.216358 20373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.216657 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.216946 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.222244 20379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.222265 20378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.222386 20373 server_base.cc:1034] running on GCE node
W20260925 02:23:03.222263 20381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.222646 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.222922 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.224086 20373 hybrid_clock.cc:648] HybridClock initialized: now 1790302983224083 us; error 36 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.226593 20373 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:23:03.227104 20373 webserver.cc:460] Webserver started at http://127.11.125.3:43753/ using document root <none> and password file <none>
I20260925 02:23:03.227344 20373 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.227416 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.227576 20373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:03.228693 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "85a7ce9995cc47eeb12c09e1ae10dc12"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "50d4866451baedd88919646e60c08638"
server_key_iv: "8be8d03eb4d18e486b81bfe9eea22749"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.229096 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "85a7ce9995cc47eeb12c09e1ae10dc12"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "50d4866451baedd88919646e60c08638"
server_key_iv: "8be8d03eb4d18e486b81bfe9eea22749"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.230484 20373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:03.231256 20389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.231459 20373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:23:03.231562 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "85a7ce9995cc47eeb12c09e1ae10dc12"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "50d4866451baedd88919646e60c08638"
server_key_iv: "8be8d03eb4d18e486b81bfe9eea22749"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.231644 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.242826 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.244874 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.245091 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.245344 20373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:03.245682 20373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:03.245745 20373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.245788 20373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:03.245816 20373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.251900 20373 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:42263
I20260925 02:23:03.251950 20503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:42263 every 8 connection(s)
I20260925 02:23:03.252278 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:23:03.254199 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20373
I20260925 02:23:03.254369 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:23:03.254573 11764 external_mini_cluster.cc:1428] Setting key 7afeac4e7b90c7f2a3334e444aeaac12
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:03.262421 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.253082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:54549 (local address 127.11.125.62:36865)
0925 02:23:03.253127 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.253129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.253340 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:23:03.253949 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.253951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.253964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.254043 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.255384 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.255978 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.257051 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.257246 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.257295 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.259751 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:03.259752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:03.259752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:03.259762 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:03.261251 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.261620 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.261621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.261621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.261671 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.261890 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.261891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.261891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.261994 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:03.262045 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:03.262291 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:23:03.262354 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:23:03.262681 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.253064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:54549)
0925 02:23:03.253212 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.253249 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.253808 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.254120 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.254124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.254999 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.255293 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.255300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.256070 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.256071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.256958 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.256961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.257320 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.257321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.257426 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.257956 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.257965 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.259666 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:03.261325 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:03.261331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:03.261333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:03.261555 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:03.261733 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:03.261733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:03.261734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:03.261836 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:03.262052 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:03.262056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:03.262177 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 02:23:03.262260 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:23:03.262400 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260925 02:23:03.263199 20504 heartbeater.cc:347] Connected to a master server at 127.11.125.62:36865
I20260925 02:23:03.263315 20504 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:03.263506 20504 heartbeater.cc:511] Master 127.11.125.62:36865 requested a full tablet report, sending...
I20260925 02:23:03.263967 20041 ts_manager.cc:194] Registered new tserver with Master: 85a7ce9995cc47eeb12c09e1ae10dc12 (127.11.125.3:42263)
I20260925 02:23:03.264487 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:54549
I20260925 02:23:03.266250 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
Entry for principal oryx/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab.
WARNING: no policy specified for oryx/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
Entry for principal oryx/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab.
WARNING: no policy specified for oryx/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
Entry for principal oryx/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab.
I20260925 02:23:03.339948 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20103
I20260925 02:23:03.345073 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20237
I20260925 02:23:03.349938 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20373
I20260925 02:23:03.354848 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20009
I20260925 02:23:03.360342 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36865
--webserver_interface=127.11.125.62
--webserver_port=38209
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:03.442247 20513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.442538 20513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.442580 20513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.443778 20513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:03.443862 20513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.443887 20513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.443912 20513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:03.443939 20513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:03.445513 20513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.445825 20513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.446079 20513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.451138 20518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.451180 20519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.451262 20521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.451431 20513 server_base.cc:1034] running on GCE node
I20260925 02:23:03.451750 20513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.452113 20513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.453347 20513 hybrid_clock.cc:648] HybridClock initialized: now 1790302983453328 us; error 49 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.456074 20513 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:23:03.456539 20513 webserver.cc:460] Webserver started at http://127.11.125.62:38209/ using document root <none> and password file <none>
I20260925 02:23:03.456797 20513 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.456890 20513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.458352 20513 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:03.459050 20528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.459232 20513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:23:03.459342 20513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "aa2fff4ff196435aabf32058e1be7ddd"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "133ff6878f9fc74c868056ceee2526e7"
server_key_iv: "d31f2802fdbf1c8af74d4c205a982bc1"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.459717 20513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.478953 20513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.480962 20513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.481153 20513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.485292 20513 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:36865
I20260925 02:23:03.485347 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:36865 every 8 connection(s)
I20260925 02:23:03.485773 20513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:23:03.487300 20581 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:23:03.487972 20581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Bootstrap starting.
I20260925 02:23:03.495225 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20513
I20260925 02:23:03.495576 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:36719
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:36865
--webserver_port=37491
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:23:03.497063 20581 log.cc:824] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:03.498780 20581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:23:03.499083 20581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Bootstrap complete.
I20260925 02:23:03.501534 20581 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:03.501899 20581 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2fff4ff196435aabf32058e1be7ddd, State: Initialized, Role: FOLLOWER
I20260925 02:23:03.502185 20581 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:03.502308 20581 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:23:03.502372 20581 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:23:03.502426 20581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:23:03.503597 20581 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:03.503751 20581 leader_election.cc:302] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2fff4ff196435aabf32058e1be7ddd; no voters:
I20260925 02:23:03.503984 20581 leader_election.cc:288] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:23:03.504068 20585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:23:03.504263 20585 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [term 2 LEADER]: Becoming Leader. State: Replica: aa2fff4ff196435aabf32058e1be7ddd, State: Running, Role: LEADER
I20260925 02:23:03.504313 20581 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:23:03.504421 20585 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } }
I20260925 02:23:03.504771 20585 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2fff4ff196435aabf32058e1be7ddd. Latest consensus state: current_term: 2 leader_uuid: "aa2fff4ff196435aabf32058e1be7ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } } }
I20260925 02:23:03.505060 20585 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:03.504976 20586 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa2fff4ff196435aabf32058e1be7ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2fff4ff196435aabf32058e1be7ddd" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36865 } } }
I20260925 02:23:03.505504 20586 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:03.505722 20594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:23:03.506366 20594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:23:03.506507 20594 catalog_manager.cc:1259] Loaded cluster ID: d746c0a8e28848d886130b0af77f189d
I20260925 02:23:03.506541 20594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:23:03.507750 20594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:23:03.507988 20594 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa2fff4ff196435aabf32058e1be7ddd: Loaded TSK: 0
I20260925 02:23:03.508205 20594 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:23:03.595170 20583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.595405 20583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.595445 20583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.596735 20583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.596808 20583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.596856 20583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:23:03.598335 20583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.1.keytab
--principal=oryx/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:36719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=37491
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.598598 20583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.598845 20583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.604790 20607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.604795 20608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.604780 20610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.605187 20583 server_base.cc:1034] running on GCE node
I20260925 02:23:03.605391 20583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.605679 20583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.606877 20583 hybrid_clock.cc:648] HybridClock initialized: now 1790302983606858 us; error 34 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.609892 20583 init.cc:377] Logged in from keytab as oryx/127.11.125.1@KRBTEST.COM (short username oryx)
I20260925 02:23:03.610320 20583 webserver.cc:460] Webserver started at http://127.11.125.1:37491/ using document root <none> and password file <none>
I20260925 02:23:03.610592 20583 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.610687 20583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.612156 20583 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:23:03.612874 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.613102 20583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:03.613211 20583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "9f71df7885764fb688cbdf45ce637631"
format_stamp: "Formatted at 2026-09-25 02:23:02 on dist-test-slave-hj7s"
server_key: "9796c6f01ec2469f183ed36d0b3ac344"
server_key_iv: "9dfd525407c7fa9bb219b8cd040e3c65"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.613574 20583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.651616 20583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.654143 20583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.654325 20583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.654593 20583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:03.654932 20583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:03.654990 20583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.655033 20583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:03.655056 20583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.661247 20583 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:36719
I20260925 02:23:03.661295 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:36719 every 8 connection(s)
I20260925 02:23:03.661650 20583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.670624 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.662375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:33699)
0925 02:23:03.662653 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.662706 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.663248 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.663760 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.663764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.664366 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.664565 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.664570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.665766 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.665767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.666578 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.666581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.667217 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.667218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.667363 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.667983 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.667993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.670390 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
W20260925 02:23:03.670687 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.662452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33699 (local address 127.11.125.62:36865)
0925 02:23:03.662664 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.662674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.662829 (+ 155us) server_negotiation.cc:407] Connection header received
0925 02:23:03.663480 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.663486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.663532 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.663689 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.664703 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.665652 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.666697 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.667107 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.667240 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.670530 (+ 3290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:33699: BlockingRecv error: recv got EOF from 127.11.125.1:33699 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
W20260925 02:23:03.670917 20732 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.125.62:36865: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
I20260925 02:23:03.671442 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20583
I20260925 02:23:03.671753 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:36255
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:36865
--webserver_port=39451
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.676167 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.671207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:48253)
0925 02:23:03.671275 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.671287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.671371 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.671522 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.671523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.671668 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.671719 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.671720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.672458 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.672459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.673229 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.673232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.673613 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.673615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.673717 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.674151 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.674155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.676058 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:23:03.676204 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.671314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:48253 (local address 127.11.125.62:36865)
0925 02:23:03.671388 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.671390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.671399 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:03.671426 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.671427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.671441 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.671517 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.671818 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.672367 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.673333 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.673528 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.673590 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.676154 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:48253: BlockingRecv error: recv got EOF from 127.11.125.1:48253 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.680974 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.676615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:44213)
0925 02:23:03.676675 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.676686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.676751 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.676936 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.676937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.677071 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.677100 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.677102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.677774 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.677775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.678273 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.678275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.678675 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.678676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.678734 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.679171 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.679176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.680859 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:23:03.681114 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.676704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:44213 (local address 127.11.125.62:36865)
0925 02:23:03.676773 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.676775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.676820 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:23:03.676839 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.676840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.676850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.676913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.677190 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.677686 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.678372 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.678561 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.678654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.681029 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:44213: BlockingRecv error: recv got EOF from 127.11.125.1:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:23:03.681298 20732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:23:03.756613 20736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.756974 20736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.757009 20736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.758168 20736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.758205 20736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.758232 20736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:23:03.759635 20736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.2.keytab
--principal=oryx/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=39451
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.759835 20736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.760018 20736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.765199 20744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.765254 20742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.765195 20741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.765421 20736 server_base.cc:1034] running on GCE node
I20260925 02:23:03.765641 20736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.765884 20736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.767030 20736 hybrid_clock.cc:648] HybridClock initialized: now 1790302983767022 us; error 30 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.776556 20736 init.cc:377] Logged in from keytab as oryx/127.11.125.2@KRBTEST.COM (short username oryx)
I20260925 02:23:03.777216 20736 webserver.cc:460] Webserver started at http://127.11.125.2:39451/ using document root <none> and password file <none>
I20260925 02:23:03.777489 20736 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.777575 20736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.778956 20736 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:23:03.779568 20751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.779744 20736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:03.779840 20736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "cca3c739882542fcbc5362f737b09ccf"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "675770a2aa35e0efddcbd02530a37f3d"
server_key_iv: "35330431ed0acd7ef97af67c6829fe17"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.780207 20736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.807096 20736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.809118 20736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.809279 20736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.809523 20736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:03.809862 20736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:03.809921 20736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.809966 20736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:03.809994 20736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.815893 20736 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36255
I20260925 02:23:03.815940 20865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36255 every 8 connection(s)
I20260925 02:23:03.816324 20736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:23:03.817677 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20736
I20260925 02:23:03.818004 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:42263
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:36865
--webserver_port=43753
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.825273 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.817094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:33865)
0925 02:23:03.817298 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.817335 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.818110 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.818408 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.818412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.819232 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.819538 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.819547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.820653 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.820654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.821522 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.821525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.822265 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.822266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.822374 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.822933 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.822941 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.825127 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
W20260925 02:23:03.825367 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.817111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:33865 (local address 127.11.125.62:36865)
0925 02:23:03.817183 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.817185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.817461 (+ 276us) server_negotiation.cc:407] Connection header received
0925 02:23:03.818224 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.818225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.818247 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.818314 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.819707 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.820532 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.821793 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.822159 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.822250 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.825278 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:33865: BlockingRecv error: recv got EOF from 127.11.125.2:33865 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:23:03.825546 20866 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.125.62:36865: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.830302 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.825818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:48503)
0925 02:23:03.825872 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.825883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.825946 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.826110 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.826111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.826241 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.826281 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.826283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.826925 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.826926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.827425 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.827427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.827805 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.827806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.827883 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.828323 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.828328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.830231 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:23:03.830430 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.825900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:48503 (local address 127.11.125.62:36865)
0925 02:23:03.825968 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.825970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.825978 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:23:03.826005 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.826007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.826016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.826103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.826370 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.826839 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.827515 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.827716 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.827804 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.830324 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:48503: BlockingRecv error: recv got EOF from 127.11.125.2:48503 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.835325 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.830622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:34211)
0925 02:23:03.830772 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.830784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.830830 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.830953 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.830954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.831057 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.831083 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.831084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.831996 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.831996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.832496 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.832498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.832987 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.832988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.833027 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.833456 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.833461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.835257 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:23:03.835476 20866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:23:03.835618 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.830705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:34211 (local address 127.11.125.62:36865)
0925 02:23:03.830750 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.830752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.830843 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:23:03.830869 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.830870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.830880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.830943 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.831153 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.831914 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.832567 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.832866 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.832952 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.835345 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:34211: BlockingRecv error: recv got EOF from 127.11.125.2:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:23:03.908052 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:03.908401 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:03.908438 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:03.910024 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:03.910084 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:03.910135 20869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:23:03.911818 20869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/krb5kdc/oryx_127.11.125.3.keytab
--principal=oryx/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:42263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=43753
--tserver_master_addrs=127.11.125.62:36865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:03.912132 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:03.912389 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:03.917855 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:03.917883 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.918100 20869 server_base.cc:1034] running on GCE node
W20260925 02:23:03.917883 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:03.918437 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:03.918659 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:03.919829 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790302983919814 us; error 29 us; skew 500 ppm
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790302983, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:03.922384 20869 init.cc:377] Logged in from keytab as oryx/127.11.125.3@KRBTEST.COM (short username oryx)
I20260925 02:23:03.922889 20869 webserver.cc:460] Webserver started at http://127.11.125.3:43753/ using document root <none> and password file <none>
I20260925 02:23:03.923142 20869 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:03.923223 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:03.924623 20869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:03.925326 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.925511 20869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.925626 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "85a7ce9995cc47eeb12c09e1ae10dc12"
format_stamp: "Formatted at 2026-09-25 02:23:03 on dist-test-slave-hj7s"
server_key: "50d4866451baedd88919646e60c08638"
server_key_iv: "8be8d03eb4d18e486b81bfe9eea22749"
server_key_version: "encryptionkey@0"
I20260925 02:23:03.926030 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:03.934593 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:03.936537 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:03.936718 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:03.937048 20869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:03.937415 20869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:03.937479 20869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.937525 20869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:03.937566 20869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:03.943674 20869 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:42263
I20260925 02:23:03.943719 20998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:42263 every 8 connection(s)
I20260925 02:23:03.944119 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.951668 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.945023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:33713)
0925 02:23:03.945208 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.945251 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.945740 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.945967 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.945970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.946563 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.946759 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.946764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.947429 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.947430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.948219 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.948220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.948580 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.948581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.948686 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.949298 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.949307 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.951486 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
W20260925 02:23:03.951673 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.945052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:33713 (local address 127.11.125.62:36865)
0925 02:23:03.945120 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.945122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.945339 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:23:03.945842 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.945843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.945852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.945913 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.946849 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.947340 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.948308 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.948494 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.948551 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.951583 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:33713: BlockingRecv error: recv got EOF from 127.11.125.3:33713 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 02:23:03.951965 20999 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:36865 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.125.62:36865: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
I20260925 02:23:03.954138 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 20869
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.957056 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.952248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:49933)
0925 02:23:03.952306 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.952317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.952355 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.952612 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.952613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.952724 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.952772 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.952774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.953391 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.953392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.953883 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.953884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.954455 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.954456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.954542 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.955029 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.955034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.956985 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:23:03.957162 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.952447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49933 (local address 127.11.125.62:36865)
0925 02:23:03.952509 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.952511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.952518 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:23:03.952539 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.952540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.952548 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.952595 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.952847 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.953326 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.954101 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.954375 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.954452 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.957074 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:49933: BlockingRecv error: recv got EOF from 127.11.125.3:49933 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:03 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:03.962159 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.957386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:35503)
0925 02:23:03.957497 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:03.957509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:03.957562 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:03.957684 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:03.957684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:03.957807 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:03.957833 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.957834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.958620 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.958621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.959410 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:03.959413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.959854 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.959855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:03.959897 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.960243 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:03.960246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:03.962068 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 02:23:03.962239 20735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:03.957452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:35503 (local address 127.11.125.62:36865)
0925 02:23:03.957516 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.957518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.957547 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:23:03.957605 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.957606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.957615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.957670 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.957934 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.958533 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.959488 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.959765 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.959843 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.962170 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:35503: BlockingRecv error: recv got EOF from 127.11.125.3:35503 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:23:03.962288 20999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:23:03.963119 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:03.954595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.11.125.62:36865)
0925 02:23:03.954734 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:03.954735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:03.954916 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:23:03.955027 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:03.955028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:03.955040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:03.955094 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:03.955776 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.956326 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.957556 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:03.957832 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:03.957881 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:03.959289 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:03.959294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:03.959294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:03.959305 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:03.961409 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.961815 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.961818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.961818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.961917 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:03.962163 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:03.962164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:03.962165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:03.962670 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:03.962812 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:03.962917 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:23:03.963004 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:04.686843 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:04.681757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:53321)
0925 02:23:04.681904 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:04.681916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:04.681967 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:04.682135 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:04.682136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:04.682303 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:04.682356 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.682358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.683109 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.683110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.684002 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.684006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.684317 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.684318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.684358 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.684815 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:04.684820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:04.686727 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260925 02:23:04.686885 21005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:04.681853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:53321 (local address 127.11.125.62:36865)
0925 02:23:04.681988 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:04.681990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:04.682004 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:23:04.682028 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:04.682029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:04.682039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:04.682107 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:04.682428 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.683004 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.684059 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.684250 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.684308 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.686810 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:53321: BlockingRecv error: recv got EOF from 127.11.125.1:53321 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:04.841308 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:04.835993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:57769)
0925 02:23:04.836168 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:04.836179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:04.836243 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:04.836443 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:04.836444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:04.836737 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:04.836792 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.836794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.837743 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.837744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.838385 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.838387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.838720 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.838721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.838763 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.839226 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:04.839231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:04.841195 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
W20260925 02:23:04.841351 21005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:04.836101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:57769 (local address 127.11.125.62:36865)
0925 02:23:04.836176 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:04.836183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:04.836245 (+ 62us) server_negotiation.cc:407] Connection header received
0925 02:23:04.836307 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:04.836307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:04.836320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:04.836384 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:04.836966 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.837631 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.838460 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.838649 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.838703 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.841284 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:57769: BlockingRecv error: recv got EOF from 127.11.125.2:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:04 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:04.967247 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:04.962750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:38127)
0925 02:23:04.962882 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:04.962901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:04.962952 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:04.963093 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:04.963093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:04.963255 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:04.963311 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.963313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.963964 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.963965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.964518 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:04.964520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.964843 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.964843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:04.964885 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.965405 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:04.965410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:04.967169 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260925 02:23:04.967357 21005 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:04.962860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:38127 (local address 127.11.125.62:36865)
0925 02:23:04.962921 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:04.962923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:04.962936 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:23:04.963004 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:04.963005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:04.963017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:04.963074 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:04.963378 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.963896 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.964595 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:04.964774 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:04.964824 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:04.967268 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:38127: BlockingRecv error: recv got EOF from 127.11.125.3:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:05.692337 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:05.687380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:58057)
0925 02:23:05.687539 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:05.687551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:05.687612 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:05.687742 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:05.687742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:05.687919 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:05.687972 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.687973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.688638 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.688639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.689255 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.689258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.689716 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.689717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.689788 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.690421 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:05.690431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:05.692269 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260925 02:23:05.692484 21009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:05.687481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:58057 (local address 127.11.125.62:36865)
0925 02:23:05.687620 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:05.687622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:05.687631 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:05.687656 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:05.687656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:05.687667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:05.687726 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:05.688045 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.688554 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.689354 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.689595 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.689681 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.692387 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:58057: BlockingRecv error: recv got EOF from 127.11.125.1:58057 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:05.846258 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:05.841867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:59245)
0925 02:23:05.842018 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:05.842031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:05.842106 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:05.842237 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:05.842238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:05.842426 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:05.842478 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.842479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.843103 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.843104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.843634 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.843636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.843977 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.843978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.844019 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.844469 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:05.844473 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:05.846194 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260925 02:23:05.846413 21009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:05.841933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:59245 (local address 127.11.125.62:36865)
0925 02:23:05.841986 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:05.841988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:05.842081 (+ 93us) server_negotiation.cc:407] Connection header received
0925 02:23:05.842161 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:05.842162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:05.842174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:05.842212 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:05.842548 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.843036 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.843713 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.843908 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.843962 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.846333 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:59245: BlockingRecv error: recv got EOF from 127.11.125.2:59245 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:05 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:05.972656 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:05.967768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:46091)
0925 02:23:05.967928 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:05.967939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:05.967996 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:05.968230 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:05.968230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:05.968403 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:05.968452 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.968453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.969225 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.969226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.969751 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:05.969752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.970076 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.970077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:05.970120 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.970535 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:05.970540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:05.972546 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
W20260925 02:23:05.972716 21009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:05.967937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:46091 (local address 127.11.125.62:36865)
0925 02:23:05.968110 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:05.968112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:05.968120 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:23:05.968144 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:05.968145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:05.968157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:05.968207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:05.968528 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.969156 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.969830 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:05.970011 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:05.970061 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:05.972651 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:46091: BlockingRecv error: recv got EOF from 127.11.125.3:46091 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:06.697692 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:06.692862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:45481)
0925 02:23:06.693044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:06.693058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:06.693129 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:06.693296 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:06.693297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:06.693458 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:06.693509 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.693510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.694165 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.694166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.694698 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.694699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.695048 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.695049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.695091 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.695534 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:06.695539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:06.697632 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":119,"threads_started":1}
W20260925 02:23:06.697854 21013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:06.692965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:45481 (local address 127.11.125.62:36865)
0925 02:23:06.693115 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:06.693117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:06.693126 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:06.693184 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:06.693184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:06.693206 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:06.693265 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:06.693579 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.694097 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.694777 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.694969 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.695031 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.697751 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:45481: BlockingRecv error: recv got EOF from 127.11.125.1:45481 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:06.851665 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:06.846738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:51981)
0925 02:23:06.846990 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:06.847002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:06.847078 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:06.847234 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:06.847235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:06.847430 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:06.847481 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.847483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.848221 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.848222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.848781 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.848783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.849227 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.849228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.849295 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.849740 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:06.849745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:06.851603 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":175,"threads_started":1}
W20260925 02:23:06.851799 21013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:06.846875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:51981 (local address 127.11.125.62:36865)
0925 02:23:06.846965 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:06.846968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:06.847075 (+ 107us) server_negotiation.cc:407] Connection header received
0925 02:23:06.847136 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:06.847137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:06.847152 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:06.847224 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:06.847567 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.848122 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.848884 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.849145 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.849228 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.851703 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:51981: BlockingRecv error: recv got EOF from 127.11.125.2:51981 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:06 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:06.978333 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:06.973235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:49269)
0925 02:23:06.973388 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:06.973407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:06.973489 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:06.973707 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:06.973708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:06.973895 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:06.973944 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.973946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.974638 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.974639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.975198 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:06.975200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.975587 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.975588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:06.975651 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.976136 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:06.976141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:06.978268 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
W20260925 02:23:06.978487 21013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:06.973264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49269 (local address 127.11.125.62:36865)
0925 02:23:06.973334 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:06.973336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:06.973474 (+ 138us) server_negotiation.cc:407] Connection header received
0925 02:23:06.973548 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:06.973550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:06.973564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:06.973618 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:06.974036 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.974548 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.975311 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:06.975501 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:06.975574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:06.978385 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:49269: BlockingRecv error: recv got EOF from 127.11.125.3:49269 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:07.703049 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:07.698253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:58239)
0925 02:23:07.698449 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:07.698461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:07.698523 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:07.698662 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:07.698663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:07.698851 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:07.698901 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.698903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.699605 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.699605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.700133 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.700135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.700500 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.700501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.700540 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.701001 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:07.701006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:07.702988 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":117,"threads_started":1}
W20260925 02:23:07.703117 21017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:07.698388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:58239 (local address 127.11.125.62:36865)
0925 02:23:07.698520 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:07.698522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:07.698531 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:07.698578 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:07.698578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:07.698589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:07.698639 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:07.698971 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.699534 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.700211 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.700432 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.700487 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.703061 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:58239: BlockingRecv error: recv got EOF from 127.11.125.1:58239 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:07.857000 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:07.852215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:46507)
0925 02:23:07.852350 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:07.852368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:07.852424 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:07.852553 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:07.852554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:07.852709 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:07.852763 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.852765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.853539 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.853540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.854073 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.854075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.854409 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.854410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.854454 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.854885 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:07.854890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:07.856873 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
W20260925 02:23:07.857092 21017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:07.852318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:46507 (local address 127.11.125.62:36865)
0925 02:23:07.852366 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:07.852372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:07.852409 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:23:07.852472 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:07.852473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:07.852483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:07.852536 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:07.852827 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.853466 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.854149 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.854343 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.854397 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.857013 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:46507: BlockingRecv error: recv got EOF from 127.11.125.2:46507 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:07 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:07.983249 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:07.978866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:48601)
0925 02:23:07.979043 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:07.979054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:07.979111 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:07.979240 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:07.979241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:07.979426 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:07.979477 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.979479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.980089 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.980090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.980620 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:07.980621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.980976 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.980977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:07.981019 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.981477 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:07.981482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:07.983188 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
W20260925 02:23:07.983392 21017 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:07.978943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:48601 (local address 127.11.125.62:36865)
0925 02:23:07.979013 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:07.979015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:07.979109 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:23:07.979154 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:07.979154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:07.979166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:07.979217 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:07.979538 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.980023 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.980689 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:07.980864 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:07.980925 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:07.983306 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:48601: BlockingRecv error: recv got EOF from 127.11.125.3:48601 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:08.709652 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:08.703728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:47685)
0925 02:23:08.703869 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:08.703884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:08.703952 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:08.704647 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:08.704648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:08.704936 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:08.705003 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.705006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.706127 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.706129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.706827 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.706830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.707183 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.707184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.707226 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.707630 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:08.707635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:08.709539 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
W20260925 02:23:08.709693 21021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:08.704232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:47685 (local address 127.11.125.62:36865)
0925 02:23:08.704495 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:08.704497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:08.704508 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:23:08.704536 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:08.704537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:08.704552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:08.704618 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:08.705145 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.706066 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.706913 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.707113 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.707175 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.709639 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:47685: BlockingRecv error: recv got EOF from 127.11.125.1:47685 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:08.869962 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:08.857482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:36357)
0925 02:23:08.857644 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:08.857656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:08.857709 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:08.857931 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:08.857932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:08.858178 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:08.858245 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.858248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.858923 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.858924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.859468 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.859470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.859862 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.859863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.859924 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.867829 (+ 7905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:08.867834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:08.869868 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
W20260925 02:23:08.870033 21021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:08.857626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:36357 (local address 127.11.125.62:36865)
0925 02:23:08.857685 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:08.857687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:08.857698 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:23:08.857792 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:08.857793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:08.857803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:08.857881 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:08.858323 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.858836 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.859560 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.859768 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.859861 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.869969 (+ 10108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:36357: BlockingRecv error: recv got EOF from 127.11.125.2:36357 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:08 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:08.988611 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:08.983784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:50067)
0925 02:23:08.983924 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:08.983939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:08.983998 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:08.984169 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:08.984170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:08.984342 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:08.984394 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.984395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.985092 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.985093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.985635 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:08.985637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.985954 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.985956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:08.985996 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.986444 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:08.986448 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:08.988549 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
W20260925 02:23:08.988715 21021 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:08.983853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:50067 (local address 127.11.125.62:36865)
0925 02:23:08.983925 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:08.983930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:08.984012 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:23:08.984081 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:08.984082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:08.984095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:08.984154 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:08.984464 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.985020 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.985710 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:08.985892 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:08.985942 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:08.988642 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:50067: BlockingRecv error: recv got EOF from 127.11.125.3:50067 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:09.719187 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:09.710203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:42763)
0925 02:23:09.710368 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:09.710382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:09.710447 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:09.710635 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:09.710636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:09.710824 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:09.710876 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.710877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.711586 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.711587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.712116 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.712118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.712495 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.712496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.712562 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.717142 (+ 4580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:09.717149 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:09.719096 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260925 02:23:09.719270 21025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:09.710269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42763 (local address 127.11.125.62:36865)
0925 02:23:09.710418 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:09.710420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:09.710428 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:23:09.710540 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:09.710541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:09.710554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:09.710625 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:09.710962 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.711488 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.712204 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.712409 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.712494 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.719197 (+ 6703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:42763: BlockingRecv error: recv got EOF from 127.11.125.1:42763 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:09.875128 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:09.870479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:47157)
0925 02:23:09.870603 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:09.870614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:09.870677 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:09.870832 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:09.870833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:09.871009 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:09.871060 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.871061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.871716 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.871717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.872234 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.872235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.872567 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.872568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.872631 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.873167 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:09.873171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:09.875063 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
W20260925 02:23:09.875264 21025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:09.870550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:47157 (local address 127.11.125.62:36865)
0925 02:23:09.870643 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:09.870644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:09.870658 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:23:09.870753 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:09.870754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:09.870766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:09.870965 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:09.871132 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.871644 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.872310 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.872495 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.872549 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.875183 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:47157: BlockingRecv error: recv got EOF from 127.11.125.2:47157 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:09 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:09.993950 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:09.989183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:43421)
0925 02:23:09.989353 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:09.989363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:09.989428 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:09.989551 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:09.989552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:09.989714 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:09.989763 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.989764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.990399 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.990400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.990914 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:09.990915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.991221 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.991222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:09.991261 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.991844 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:09.991849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:09.993851 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
W20260925 02:23:09.993997 21025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:09.989227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:43421 (local address 127.11.125.62:36865)
0925 02:23:09.989310 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:09.989313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:09.989426 (+ 113us) server_negotiation.cc:407] Connection header received
0925 02:23:09.989472 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:09.989473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:09.989486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:09.989536 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:09.989834 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.990333 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.990981 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:09.991160 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:09.991206 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:09.993933 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:43421: BlockingRecv error: recv got EOF from 127.11.125.3:43421 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:10.724620 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:10.719773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:56427)
0925 02:23:10.719939 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:10.719951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:10.720005 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:10.720139 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:10.720140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:10.720319 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:10.720369 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.720371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.721058 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.721059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.721600 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.721602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.721939 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.721940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.721983 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.722532 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:10.722539 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:10.724549 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
W20260925 02:23:10.724730 21029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:10.719870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:56427 (local address 127.11.125.62:36865)
0925 02:23:10.719984 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:10.719986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:10.719995 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:10.720051 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:10.720052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:10.720063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:10.720120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:10.720436 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.720975 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.721671 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.721873 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.721923 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.724651 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:56427: BlockingRecv error: recv got EOF from 127.11.125.1:56427 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:10.880402 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:10.875558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:38519)
0925 02:23:10.875730 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:10.875740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:10.875796 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:10.875932 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:10.875933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:10.876114 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:10.876166 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.876168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.876847 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.876848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.877622 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.877626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.877977 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.877978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.878033 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.878497 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:10.878502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:10.880341 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
W20260925 02:23:10.880502 21029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:10.875634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:38519 (local address 127.11.125.62:36865)
0925 02:23:10.875702 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:10.875704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:10.875800 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:23:10.875859 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:10.875860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:10.875871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:10.875910 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:10.876227 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.876735 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.877711 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.877907 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.877964 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.880435 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:38519: BlockingRecv error: recv got EOF from 127.11.125.2:38519 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:10 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:10.998956 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:10.994565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:52809)
0925 02:23:10.994706 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:10.994718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:10.994799 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:10.994950 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:10.994951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:10.995115 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:10.995167 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.995169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.995832 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.995833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.996343 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:10.996344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.996660 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.996660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:10.996700 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.997146 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:10.997150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:10.998889 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260925 02:23:10.999090 21029 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:10.994615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:52809 (local address 127.11.125.62:36865)
0925 02:23:10.994683 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:10.994685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:10.994784 (+ 99us) server_negotiation.cc:407] Connection header received
0925 02:23:10.994851 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:10.994852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:10.994865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:10.994919 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:10.995227 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.995749 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.996413 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:10.996592 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:10.996641 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:10.999004 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:52809: BlockingRecv error: recv got EOF from 127.11.125.3:52809 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:11.730247 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:11.725275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:55481)
0925 02:23:11.725418 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:11.725429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:11.725480 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:11.725701 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:11.725703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:11.725947 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:11.726000 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:11.726002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.726701 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.726702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:11.727242 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:11.727244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.727658 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.727659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:11.727720 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:11.728213 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:11.728219 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:11.730186 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260925 02:23:11.730412 21033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:11.725394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:55481 (local address 127.11.125.62:36865)
0925 02:23:11.725546 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:11.725548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:11.725557 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:11.725583 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:11.725584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:11.725595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:11.725689 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:11.726086 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.726604 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.727346 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.727550 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.727656 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:11.730320 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:55481: BlockingRecv error: recv got EOF from 127.11.125.1:55481 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:11.885957 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:11.881072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:49481)
0925 02:23:11.881248 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:11.881265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:11.881340 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:11.881485 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:11.881486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:11.881656 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:11.881709 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:11.881711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.882491 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.882492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:11.883037 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:11.883039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.883397 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.883398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:11.883463 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:11.884045 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:11.884050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:11.885884 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
W20260925 02:23:11.886034 21033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:11.881144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:49481 (local address 127.11.125.62:36865)
0925 02:23:11.881220 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:11.881222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:11.881325 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:23:11.881397 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:11.881397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:11.881410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:11.881461 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:11.881770 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.882420 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.883103 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:11.883307 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:11.883361 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:11.885983 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:49481: BlockingRecv error: recv got EOF from 127.11.125.2:49481 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:11 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:12.003903 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:11.999398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:49465)
0925 02:23:11.999518 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:11.999538 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:11.999607 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:11.999735 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:11.999735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:11.999903 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:11.999954 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:11.999955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.000597 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.000597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.001199 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:12.001201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.001533 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.001534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.001574 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.002075 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:12.002080 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:12.003842 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
W20260925 02:23:12.004040 21033 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:11.999482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49465 (local address 127.11.125.62:36865)
0925 02:23:11.999532 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:11.999535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:11.999590 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:23:11.999658 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:11.999659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:11.999671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:11.999708 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:12.000023 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.000526 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.001273 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.001467 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.001521 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.003953 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:49465: BlockingRecv error: recv got EOF from 127.11.125.3:49465 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 02:23:12 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:12 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:12.735468 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:12.730700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:41749)
0925 02:23:12.730837 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:12.730851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:12.730934 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:12.731072 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:12.731073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:12.731239 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:12.731322 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:12.731324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.731995 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.731996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.732556 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:12.732558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.732954 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.732955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.733011 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.733535 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:12.733540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:12.735397 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
W20260925 02:23:12.735623 21037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:12.730799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:41749 (local address 127.11.125.62:36865)
0925 02:23:12.730938 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:12.730940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:12.730966 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:23:12.730989 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:12.730990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:12.731001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:12.731051 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:12.731394 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.731913 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.732633 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.732841 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.732892 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.735524 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:41749: BlockingRecv error: recv got EOF from 127.11.125.1:41749 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
Sep 25 02:23:12 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:12 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:12.891598 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:12.886483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:38793)
0925 02:23:12.886665 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:12.886678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:12.886779 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:12.886971 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:12.886972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:12.887166 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:12.887214 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:12.887215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.887942 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.887944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.888481 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:12.888483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.888869 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.888870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:12.888965 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.889460 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:12.889464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:12.891485 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
W20260925 02:23:12.891638 21037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:12.886597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:38793 (local address 127.11.125.62:36865)
0925 02:23:12.886672 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:12.886674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:12.886758 (+ 84us) server_negotiation.cc:407] Connection header received
0925 02:23:12.886861 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:12.886862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:12.886876 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:12.886947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:12.887321 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.887845 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.888572 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:12.888773 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:12.888867 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:12.891572 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:38793: BlockingRecv error: recv got EOF from 127.11.125.2:38793 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:13.009920 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:13.004505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:38621)
0925 02:23:13.004661 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:13.004672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:13.004742 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:13.004959 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:13.004961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:13.005242 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:13.005332 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.005334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.006380 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.006382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.006936 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.006937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.007328 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.007329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.007391 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.007864 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:13.007869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:13.009812 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
W20260925 02:23:13.010061 21037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:13.004554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:38621 (local address 127.11.125.62:36865)
0925 02:23:13.004642 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:13.004644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:13.004729 (+ 85us) server_negotiation.cc:407] Connection header received
0925 02:23:13.004830 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:13.004831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:13.004846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:13.004943 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:13.005421 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.006284 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.007032 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.007247 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.007326 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.009954 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:38621: BlockingRecv error: recv got EOF from 127.11.125.3:38621 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:13.740641 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:13.735963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:41185)
0925 02:23:13.736101 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:13.736114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:13.736179 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:13.736316 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:13.736316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:13.736476 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:13.736527 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.736528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.737224 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.737225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.737763 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.737765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.738210 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.738211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.738252 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.738803 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:13.738808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:13.740581 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260925 02:23:13.740830 21041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:13.736022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:41185 (local address 127.11.125.62:36865)
0925 02:23:13.736154 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:13.736155 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:13.736166 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:23:13.736231 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:13.736232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:13.736246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:13.736293 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:13.736592 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.737155 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.737924 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.738136 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.738201 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.740737 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:41185: BlockingRecv error: recv got EOF from 127.11.125.1:41185 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:13 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:23:13.903047 21041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:13.892249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:50099 (local address 127.11.125.62:36865)
0925 02:23:13.892344 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:13.892346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:13.892414 (+ 68us) server_negotiation.cc:407] Connection header received
0925 02:23:13.892491 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:13.892492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:13.892506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:13.892555 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:13.892892 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.893431 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.894169 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.894379 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.894461 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.902961 (+ 8500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:50099: BlockingRecv error: recv got EOF from 127.11.125.2:50099 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:23:13.903074 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:13.892145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:50099)
0925 02:23:13.892334 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:13.892348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:13.892426 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:13.892568 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:13.892568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:13.892754 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:13.892813 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.892815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.893527 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.893529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.894078 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:13.894079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:13.894459 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:13.894460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:13.894522 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:13.899567 (+ 5045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:13.899574 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:13.902886 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:14.015341 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:14.010509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:37121)
0925 02:23:14.010682 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:14.010696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:14.010769 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:14.010932 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:14.010933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:14.011107 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:14.011157 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.011159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.011885 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.011886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.012423 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.012425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.012743 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.012744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.012784 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.013264 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:14.013269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:14.015275 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
W20260925 02:23:14.015478 21041 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:14.010656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:37121 (local address 127.11.125.62:36865)
0925 02:23:14.010726 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:14.010727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:14.010754 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:23:14.010844 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:14.010845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:14.010857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:14.010906 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:14.011242 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.011791 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.012496 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.012679 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.012730 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.015394 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:37121: BlockingRecv error: recv got EOF from 127.11.125.3:37121 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:14.745688 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:14.741120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:33147)
0925 02:23:14.741257 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:14.741268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:14.741326 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:14.741488 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:14.741489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:14.741659 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:14.741710 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.741712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.742369 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.742370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.742930 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.742931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.743264 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.743265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.743308 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.743743 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:14.743748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:14.745619 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260925 02:23:14.745815 21045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:14.741166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33147 (local address 127.11.125.62:36865)
0925 02:23:14.741334 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:14.741336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:14.741344 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:23:14.741366 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:14.741367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:14.741377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:14.741439 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:14.741784 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.742277 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.743008 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.743199 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.743250 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.745716 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:33147: BlockingRecv error: recv got EOF from 127.11.125.1:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:14 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:14.908870 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:14.903750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:50805)
0925 02:23:14.903940 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:14.903952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:14.904016 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:14.904183 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:14.904184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:14.904398 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:14.904470 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.904472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.905350 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.905351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.905988 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:14.905990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.906391 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.906393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:14.906438 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.906953 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:14.906958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:14.908800 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
W20260925 02:23:14.909034 21045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:14.903811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:50805 (local address 127.11.125.62:36865)
0925 02:23:14.903859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:14.903861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:14.903995 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:23:14.904077 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:14.904078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:14.904090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:14.904159 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:14.904547 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.905240 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.906101 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:14.906319 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:14.906383 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:14.908949 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:50805: BlockingRecv error: recv got EOF from 127.11.125.2:50805 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:15.020539 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:15.015836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:52813)
0925 02:23:15.016117 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:15.016129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:15.016211 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:15.016373 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:15.016374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:15.016545 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:15.016593 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.016594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.017353 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.017353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.017899 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.017901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.018211 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.018212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.018252 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.018691 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:15.018696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:15.020471 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":170,"threads_started":1}
W20260925 02:23:15.020614 21045 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:15.016065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:52813 (local address 127.11.125.62:36865)
0925 02:23:15.016125 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:15.016129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:15.016214 (+ 85us) server_negotiation.cc:407] Connection header received
0925 02:23:15.016259 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:15.016260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:15.016280 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:15.016345 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:15.016661 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.017251 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.017968 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.018147 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.018198 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.020565 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:52813: BlockingRecv error: recv got EOF from 127.11.125.3:52813 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:15.751266 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:15.746223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:43247)
0925 02:23:15.746372 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:15.746385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:15.746446 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:15.746653 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:15.746654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:15.746850 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:15.746899 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.746901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.747604 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.747605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.748181 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.748183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.748654 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.748655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.748706 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.749279 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:15.749284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:15.751182 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":113,"threads_started":1}
W20260925 02:23:15.751381 21049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:15.746311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:43247 (local address 127.11.125.62:36865)
0925 02:23:15.746468 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:15.746470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:15.746480 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:23:15.746504 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:15.746505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:15.746517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:15.746655 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:15.746985 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.747529 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.748283 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.748566 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.748636 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.751290 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:43247: BlockingRecv error: recv got EOF from 127.11.125.1:43247 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:15 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:15.913946 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:15.909418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:57975)
0925 02:23:15.909562 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:15.909581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:15.909635 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:15.909767 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:15.909768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:15.909937 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:15.909990 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.909991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.910629 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.910630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.911156 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:15.911157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.911514 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.911515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:15.911558 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.911993 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:15.911997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:15.913886 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260925 02:23:15.914088 21049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:15.909496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:57975 (local address 127.11.125.62:36865)
0925 02:23:15.909548 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:15.909550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:15.909647 (+ 97us) server_negotiation.cc:407] Connection header received
0925 02:23:15.909691 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:15.909692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:15.909705 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:15.909750 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:15.910051 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.910554 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.911239 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:15.911444 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:15.911503 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:15.914005 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:57975: BlockingRecv error: recv got EOF from 127.11.125.2:57975 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:16.026087 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:16.021178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:35395)
0925 02:23:16.021375 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:16.021387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:16.021450 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:16.021579 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:16.021580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:16.021754 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:16.021803 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.021804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.022438 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.022439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.023274 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.023277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.023616 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.023617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.023656 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.024077 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:16.024082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:16.026000 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
W20260925 02:23:16.026172 21049 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:16.021238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:35395 (local address 127.11.125.62:36865)
0925 02:23:16.021338 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:16.021340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:16.021450 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:23:16.021494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:16.021495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:16.021507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:16.021558 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:16.021864 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.022370 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.023353 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.023558 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.023610 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.026093 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:35395: BlockingRecv error: recv got EOF from 127.11.125.3:35395 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:16.756443 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:16.751823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:59073)
0925 02:23:16.751997 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:16.752008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:16.752064 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:16.752198 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:16.752198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:16.752372 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:16.752425 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.752426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.753147 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.753148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.753696 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.753698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.754028 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.754029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.754071 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.754503 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:16.754508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:16.756344 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
W20260925 02:23:16.756502 21053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:16.751919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:59073 (local address 127.11.125.62:36865)
0925 02:23:16.752039 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:16.752041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:16.752050 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:16.752125 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:16.752126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:16.752136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:16.752191 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:16.752486 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.753054 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.753772 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.753965 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.754014 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.756428 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:59073: BlockingRecv error: recv got EOF from 127.11.125.1:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:16 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:16.919072 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:16.914409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:34471)
0925 02:23:16.914552 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:16.914564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:16.914619 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:16.914744 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:16.914745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:16.914928 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:16.914981 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.914982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.915624 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.915625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.916221 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:16.916223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.916562 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.916563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:16.916603 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.917161 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:16.917165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:16.918972 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
W20260925 02:23:16.919123 21053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:16.914541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:34471 (local address 127.11.125.62:36865)
0925 02:23:16.914595 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:16.914597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:16.914606 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:16.914665 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:16.914665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:16.914677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:16.914722 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:16.915047 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.915552 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.916305 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:16.916496 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:16.916549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:16.919058 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:34471: BlockingRecv error: recv got EOF from 127.11.125.2:34471 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:17.031697 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:17.026592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:40379)
0925 02:23:17.026770 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:17.026782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:17.026888 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:17.027049 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:17.027050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:17.027245 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:17.027295 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.027297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.027987 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.027988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.028520 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.028522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.028997 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.028998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.029069 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.029528 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:17.029533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:17.031637 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260925 02:23:17.031816 21053 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:17.026684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:40379 (local address 127.11.125.62:36865)
0925 02:23:17.026765 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:17.026767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:17.026868 (+ 101us) server_negotiation.cc:407] Connection header received
0925 02:23:17.026955 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:17.026956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:17.026974 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:17.027043 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:17.027381 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.027895 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.028608 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.028876 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.028995 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.031723 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:40379: BlockingRecv error: recv got EOF from 127.11.125.3:40379 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:17.761896 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:17.757003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:51089)
0925 02:23:17.757160 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:17.757171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:17.757230 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:17.757435 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:17.757437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:17.757635 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:17.757717 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.757720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.758528 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.758529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.759059 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.759060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.759390 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.759391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.759433 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.759902 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:17.759906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:17.761836 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
W20260925 02:23:17.762022 21057 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:17.757101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:51089 (local address 127.11.125.62:36865)
0925 02:23:17.757244 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:17.757246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:17.757287 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:23:17.757316 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:17.757317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:17.757331 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:17.757400 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:17.757876 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.758455 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.759132 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.759324 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.759375 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.761936 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:51089: BlockingRecv error: recv got EOF from 127.11.125.1:51089 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:17 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:17.924885 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:17.919634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:45221)
0925 02:23:17.919869 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:17.919880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:17.919969 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:17.920184 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:17.920186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:17.920419 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:17.920478 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.920481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.921419 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.921427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.922059 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:17.922062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.922442 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.922443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:17.922501 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.922966 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:17.922972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:17.924770 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
W20260925 02:23:17.924952 21057 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:17.919733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:45221 (local address 127.11.125.62:36865)
0925 02:23:17.919819 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:17.919822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:17.919947 (+ 125us) server_negotiation.cc:407] Connection header received
0925 02:23:17.920105 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:17.920106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:17.920120 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:17.920174 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:17.920564 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.921326 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.922169 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:17.922366 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:17.922444 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:17.924854 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:45221: BlockingRecv error: recv got EOF from 127.11.125.2:45221 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.3@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:18.037176 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:18.032217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.3:49569)
0925 02:23:18.032408 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:18.032420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:18.032489 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:18.032651 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:18.032652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:18.032858 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:18.032951 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.032953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.033630 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.033631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.034170 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.034171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.034625 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.034626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.034686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.035232 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:18.035238 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:18.037067 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
W20260925 02:23:18.037225 21057 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:18.032309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49569 (local address 127.11.125.62:36865)
0925 02:23:18.032384 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:18.032386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:18.032477 (+ 91us) server_negotiation.cc:407] Connection header received
0925 02:23:18.032552 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:18.032553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:18.032568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:18.032644 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:18.033019 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.033543 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.034255 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.034550 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.034615 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.037158 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.3:49569: BlockingRecv error: recv got EOF from 127.11.125.3:49569 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.1@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:18.767094 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:18.762472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.1:44551)
0925 02:23:18.762696 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:18.762709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:18.762778 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:18.762990 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:18.762991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:18.763168 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:18.763220 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.763222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.763905 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.763906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.764441 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.764443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.764800 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.764801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.764844 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.765343 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:18.765348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:18.767026 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":132,"threads_started":1}
W20260925 02:23:18.767204 21061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:18.762644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:44551 (local address 127.11.125.62:36865)
0925 02:23:18.762776 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:18.762778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:18.762787 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:23:18.762873 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:18.762874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:18.762889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:18.762950 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:18.763285 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.763835 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.764520 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.764720 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.764776 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.767126 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.1:44551: BlockingRecv error: recv got EOF from 127.11.125.1:44551 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:23:18 dist-test-slave-hj7s krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.125.2@KRBTEST.COM for oryx/127.11.125.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:23:18.943163 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:18.925462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:36865 (local address 127.11.125.2:57197)
0925 02:23:18.925618 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:18.925635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:18.925697 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:18.925828 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:18.925829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:18.925986 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:18.926039 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.926041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.926679 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.926680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.927207 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:18.927209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.927541 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.927542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:18.927583 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.941216 (+ 13633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:18.941223 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:18.943081 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.125.62:36865: Server oryx/127.11.125.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260925 02:23:18.943384 21061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:18.925559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:57197 (local address 127.11.125.62:36865)
0925 02:23:18.925642 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:18.925644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:18.925680 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:23:18.925746 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:18.925747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:18.925758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:18.925822 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:18.926104 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.926605 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.927278 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:18.927472 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:18.927523 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:18.943218 (+ 15695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.125.2:57197: BlockingRecv error: recv got EOF from 127.11.125.2:57197 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:23:18.964736 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20583
I20260925 02:23:18.969568 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20736
I20260925 02:23:18.974510 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20869
I20260925 02:23:18.978802 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 20513
2026-09-25T02:23:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:18Z Disabled control of system clock
I20260925 02:23:19.007560 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42787
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:23:19.088609 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.088828 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.088850 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.090087 21066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:19.090131 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.090144 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.090157 21066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:19.090212 21066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:19.091594 21066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:42787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.091866 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.092181 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.097286 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.097343 21066 server_base.cc:1034] running on GCE node
W20260925 02:23:19.097369 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.097370 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.097681 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.097992 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.099164 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790302999099144 us; error 39 us; skew 500 ppm
I20260925 02:23:19.100481 21066 webserver.cc:460] Webserver started at http://127.11.125.62:35693/ using document root <none> and password file <none>
I20260925 02:23:19.100757 21066 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.100862 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.101049 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.102140 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "65b20c93201b4b479eae9b180e619538"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "413bb509f84afa8c73738d4de389e9fe"
server_key_iv: "cf09fa0ad63c8bbe73db82c98ad97578"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.102526 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "65b20c93201b4b479eae9b180e619538"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "413bb509f84afa8c73738d4de389e9fe"
server_key_iv: "cf09fa0ad63c8bbe73db82c98ad97578"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.103883 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:23:19.104583 21080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.104754 21066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:19.104871 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "65b20c93201b4b479eae9b180e619538"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "413bb509f84afa8c73738d4de389e9fe"
server_key_iv: "cf09fa0ad63c8bbe73db82c98ad97578"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.104991 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.115187 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.115489 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.115644 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.119724 21066 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:42787
I20260925 02:23:19.119757 21132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:42787 every 8 connection(s)
I20260925 02:23:19.120110 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:23:19.120760 21133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:19.121969 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21066
I20260925 02:23:19.122087 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:23:19.122289 11764 external_mini_cluster.cc:1428] Setting key 6b119f23d260d0a65959a767c9a3c3d4
I20260925 02:23:19.124315 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538: Bootstrap starting.
I20260925 02:23:19.125007 21133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:19.125324 21133 log.cc:824] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:19.126083 21133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538: No bootstrap required, opened a new log
I20260925 02:23:19.127954 21133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } }
I20260925 02:23:19.128083 21133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:19.128105 21133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b20c93201b4b479eae9b180e619538, State: Initialized, Role: FOLLOWER
I20260925 02:23:19.128176 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.122874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.11.125.62:42787)
0925 02:23:19.123065 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.123073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.123098 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:23:19.123502 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.123509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.123567 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.123685 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.123947 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.124884 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.125972 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.126453 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.126545 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.127237 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.127241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.127241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.127431 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.127463 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.127474 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.127526 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.127625 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.127706 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260925 02:23:19.128296 21133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } }
I20260925 02:23:19.128392 21133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:23:19.128432 21133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:23:19.128479 21133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:19.129191 21133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } }
I20260925 02:23:19.129323 21133 leader_election.cc:302] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b20c93201b4b479eae9b180e619538; no voters:
I20260925 02:23:19.129483 21133 leader_election.cc:288] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:23:19.129529 21138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:19.129716 21138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [term 1 LEADER]: Becoming Leader. State: Replica: 65b20c93201b4b479eae9b180e619538, State: Running, Role: LEADER
I20260925 02:23:19.129786 21133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:23:19.129873 21138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } }
I20260925 02:23:19.130286 21139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b20c93201b4b479eae9b180e619538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } } }
I20260925 02:23:19.130321 21140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b20c93201b4b479eae9b180e619538. Latest consensus state: current_term: 1 leader_uuid: "65b20c93201b4b479eae9b180e619538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b20c93201b4b479eae9b180e619538" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42787 } } }
I20260925 02:23:19.130388 21139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:19.130404 21140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:19.130592 21147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:23:19.131124 21147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:23:19.133059 21147 catalog_manager.cc:1347] Generated new cluster ID: e5abea8bc64f41bfb44b4bde30d57b59
I20260925 02:23:19.133114 21147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:23:19.139743 21147 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:23:19.140338 21147 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:23:19.148069 21147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b20c93201b4b479eae9b180e619538: Generated new TSK 0
I20260925 02:23:19.148277 21147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:23:19.153185 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--builtin_ntp_servers=127.11.125.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:23:19.252429 21158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.252627 21158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.252650 21158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.253859 21158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.253902 21158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.253928 21158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:23:19.255270 21158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.255520 21158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.255847 21158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.261852 21163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.261849 21164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.262151 21166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.262338 21158 server_base.cc:1034] running on GCE node
I20260925 02:23:19.262580 21158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.262849 21158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.264051 21158 hybrid_clock.cc:648] HybridClock initialized: now 1790302999264031 us; error 34 us; skew 500 ppm
I20260925 02:23:19.265515 21158 webserver.cc:460] Webserver started at http://127.11.125.1:34283/ using document root <none> and password file <none>
I20260925 02:23:19.265775 21158 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.265825 21158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.265959 21158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.267033 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "12557188f245420fb3a15f542dbce6ed"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "1a4ac4cc41c6c5c51448bcecb740b72b"
server_key_iv: "066153969010fceec5c4023f7998084f"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.267407 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "12557188f245420fb3a15f542dbce6ed"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "1a4ac4cc41c6c5c51448bcecb740b72b"
server_key_iv: "066153969010fceec5c4023f7998084f"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.268815 21158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:23:19.269657 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.269816 21158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:23:19.269922 21158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "12557188f245420fb3a15f542dbce6ed"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "1a4ac4cc41c6c5c51448bcecb740b72b"
server_key_iv: "066153969010fceec5c4023f7998084f"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.269997 21158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.293273 21158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.293634 21158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.293797 21158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.294049 21158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:19.294409 21158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:19.294463 21158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.294500 21158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:19.294529 21158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.300632 21158 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:36323
I20260925 02:23:19.300683 21287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:36323 every 8 connection(s)
I20260925 02:23:19.301059 21158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:23:19.306423 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.301773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:47405 (local address 127.11.125.62:42787)
0925 02:23:19.301831 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.301833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.302074 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:23:19.302636 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.302637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.302649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.302697 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.303382 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.303918 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.304959 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.305168 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.305249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.306109 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.306110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.306110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.306115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.306145 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.306149 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.306211 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.306312 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.306345 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:23:19.306669 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.301731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42787 (local address 127.11.125.1:47405)
0925 02:23:19.301919 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:19.301966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:19.302521 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:19.302767 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:19.302772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:19.302782 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:19.303277 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.303281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.304026 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.304028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.304834 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.304836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.305260 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.305261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.305402 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.305948 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:19.305949 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:19.305978 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:19.306001 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:19.306213 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:19.306220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:19.306224 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:23:19.306314 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:23:19.306419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260925 02:23:19.307242 21288 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42787
I20260925 02:23:19.307323 21288 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:19.307524 21288 heartbeater.cc:511] Master 127.11.125.62:42787 requested a full tablet report, sending...
I20260925 02:23:19.308037 21097 ts_manager.cc:194] Registered new tserver with Master: 12557188f245420fb3a15f542dbce6ed (127.11.125.1:36323)
I20260925 02:23:19.308656 21097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.1:47405
I20260925 02:23:19.309132 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21158
I20260925 02:23:19.309207 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:23:19.309417 11764 external_mini_cluster.cc:1428] Setting key 3060eee66becefef3e6296c69d6a9d01
I20260925 02:23:19.310189 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--builtin_ntp_servers=127.11.125.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:23:19.391706 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.391914 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.391934 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.393164 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.393206 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.393252 21291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:23:19.394589 21291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.394853 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.395164 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.400341 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.400283 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.400393 21291 server_base.cc:1034] running on GCE node
W20260925 02:23:19.400295 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.400713 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.401062 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.402221 21291 hybrid_clock.cc:648] HybridClock initialized: now 1790302999402208 us; error 47 us; skew 500 ppm
I20260925 02:23:19.403447 21291 webserver.cc:460] Webserver started at http://127.11.125.2:44165/ using document root <none> and password file <none>
I20260925 02:23:19.403713 21291 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.403808 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.403973 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.405126 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "36ae3be249744fcabcbb23faea6585db"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "402a2234cbbb639e608c213d2fe3114d"
server_key_iv: "9ab97bdf6f58656d830fbedfa3103337"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.405493 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "36ae3be249744fcabcbb23faea6585db"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "402a2234cbbb639e608c213d2fe3114d"
server_key_iv: "9ab97bdf6f58656d830fbedfa3103337"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.406806 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:19.407632 21305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.407823 21291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:19.407924 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "36ae3be249744fcabcbb23faea6585db"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "402a2234cbbb639e608c213d2fe3114d"
server_key_iv: "9ab97bdf6f58656d830fbedfa3103337"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.408012 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.417410 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.417711 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.417873 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.418110 21291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:19.418444 21291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:19.418504 21291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.418540 21291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:19.418568 21291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.424507 21291 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:38969
I20260925 02:23:19.424553 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:38969 every 8 connection(s)
I20260925 02:23:19.424886 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:23:19.430410 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.425715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:43321 (local address 127.11.125.62:42787)
0925 02:23:19.425764 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.425766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.425999 (+ 233us) server_negotiation.cc:407] Connection header received
0925 02:23:19.426575 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.426577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.426591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.426638 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.427376 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.427892 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.428892 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.429121 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.429201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.430131 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.430132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.430132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.430138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.430154 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.430158 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.430211 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.430324 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.430351 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:23:19.430613 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.425676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42787 (local address 127.11.125.2:43321)
0925 02:23:19.425859 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:19.425904 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:19.426459 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:19.426718 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:19.426722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:19.426731 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:19.427253 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.427258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.427995 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.427996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.428793 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.428796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.429211 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.429212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.429361 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.429952 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:19.429952 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:19.430002 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:19.430026 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:19.430228 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:19.430234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:19.430237 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:23:19.430311 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:23:19.430398 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260925 02:23:19.431123 21420 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42787
I20260925 02:23:19.431202 21420 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:19.431428 21420 heartbeater.cc:511] Master 127.11.125.62:42787 requested a full tablet report, sending...
I20260925 02:23:19.431816 21097 ts_manager.cc:194] Registered new tserver with Master: 36ae3be249744fcabcbb23faea6585db (127.11.125.2:38969)
I20260925 02:23:19.432147 21097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.2:43321
I20260925 02:23:19.434985 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21291
I20260925 02:23:19.435071 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:23:19.435221 11764 external_mini_cluster.cc:1428] Setting key 6a00081ee19149b44aa60b1705c93b67
I20260925 02:23:19.436163 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--builtin_ntp_servers=127.11.125.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:23:19.520355 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.520643 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.520687 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.522056 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.522142 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.522202 21423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:23:19.523754 21423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.524062 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.524435 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.530272 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.530280 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.530268 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.530740 21423 server_base.cc:1034] running on GCE node
I20260925 02:23:19.530992 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.531252 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.532440 21423 hybrid_clock.cc:648] HybridClock initialized: now 1790302999532414 us; error 53 us; skew 500 ppm
I20260925 02:23:19.533838 21423 webserver.cc:460] Webserver started at http://127.11.125.3:45473/ using document root <none> and password file <none>
I20260925 02:23:19.534114 21423 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.534188 21423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.534340 21423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.535459 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "386cb4ee004a48eb9a34b38d4ebde765"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "8ca1f33a87bd0f8c411a2c10e3ea1e90"
server_key_iv: "a7e46eade1bf2fa2320b62a5df71b86a"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.535847 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "386cb4ee004a48eb9a34b38d4ebde765"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "8ca1f33a87bd0f8c411a2c10e3ea1e90"
server_key_iv: "a7e46eade1bf2fa2320b62a5df71b86a"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.537380 21423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:23:19.538187 21437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.538373 21423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:19.538468 21423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "386cb4ee004a48eb9a34b38d4ebde765"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "8ca1f33a87bd0f8c411a2c10e3ea1e90"
server_key_iv: "a7e46eade1bf2fa2320b62a5df71b86a"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.538555 21423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.555781 21423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.556142 21423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.556303 21423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.556571 21423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:19.557003 21423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:19.557058 21423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.557102 21423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:19.557130 21423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.562963 21423 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:43825
I20260925 02:23:19.563009 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:43825 every 8 connection(s)
I20260925 02:23:19.563342 21423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:23:19.569283 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.564162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:37889 (local address 127.11.125.62:42787)
0925 02:23:19.564259 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.564262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.564483 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:23:19.565241 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.565242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.565257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.565305 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.566060 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.566617 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.567724 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.567936 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.568017 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.568848 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.568850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.568850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.568855 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.568873 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.568883 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.568992 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.569164 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.569194 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:23:19.569499 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.564053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42787 (local address 127.11.125.3:37889)
0925 02:23:19.564335 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:19.564383 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:19.565110 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:19.565394 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:19.565398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:19.565424 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:19.565960 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.565965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.566725 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.566726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.567613 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:19.567615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:19.568027 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:19.568028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:19.568182 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:19.568695 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:19.568695 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:19.568725 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:19.568749 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:19.568997 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:19.569004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:19.569008 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:23:19.569164 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:23:19.569274 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260925 02:23:19.569988 21552 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42787
I20260925 02:23:19.570101 21552 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:19.570307 21552 heartbeater.cc:511] Master 127.11.125.62:42787 requested a full tablet report, sending...
I20260925 02:23:19.570708 21097 ts_manager.cc:194] Registered new tserver with Master: 386cb4ee004a48eb9a34b38d4ebde765 (127.11.125.3:43825)
I20260925 02:23:19.571087 21097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.3:37889
I20260925 02:23:19.571334 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21423
I20260925 02:23:19.571449 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:23:19.571650 11764 external_mini_cluster.cc:1428] Setting key a68bd910ad9725a66b30063ac9c034ba
I20260925 02:23:19.572491 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:23:19.574259 21136 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:19.573868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.11.125.62:42787)
0925 02:23:19.573904 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.573905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.574029 (+ 124us) server_negotiation.cc:407] Connection header received
0925 02:23:19.574069 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.574070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.574077 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.574128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.574218 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34962: BlockingRecv error: recv got EOF from 127.0.0.1:34962 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:23:19.574405 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21158
I20260925 02:23:19.580720 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21291
I20260925 02:23:19.585345 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21423
I20260925 02:23:19.589802 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21066
2026-09-25T02:23:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (610 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:19Z Disabled control of system clock
I20260925 02:23:19.617421 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42609
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:23:19.704077 21563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.704293 21563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.704317 21563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.705539 21563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:19.705592 21563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.705605 21563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.705618 21563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:19.705629 21563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:19.706933 21563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:42609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.707139 21563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.707396 21563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.712381 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.712399 21569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.712417 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.712643 21563 server_base.cc:1034] running on GCE node
I20260925 02:23:19.712939 21563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.713245 21563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.714411 21563 hybrid_clock.cc:648] HybridClock initialized: now 1790302999714399 us; error 41 us; skew 500 ppm
I20260925 02:23:19.715696 21563 webserver.cc:460] Webserver started at http://127.11.125.62:46349/ using document root <none> and password file <none>
I20260925 02:23:19.715957 21563 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.716030 21563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.716179 21563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.717356 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "2a590fee63f04d3a968f39e6d9dab301"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "09fcef98a397649fe4754a09d3b315a0"
server_key_iv: "d7b108a2523a88bd2654bbef3b0320b3"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.717715 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "2a590fee63f04d3a968f39e6d9dab301"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "09fcef98a397649fe4754a09d3b315a0"
server_key_iv: "d7b108a2523a88bd2654bbef3b0320b3"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.718993 21563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:23:19.719683 21577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.719866 21563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:19.719973 21563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "2a590fee63f04d3a968f39e6d9dab301"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "09fcef98a397649fe4754a09d3b315a0"
server_key_iv: "d7b108a2523a88bd2654bbef3b0320b3"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.720048 21563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.723443 21563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.723704 21563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.723851 21563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.727712 21563 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:42609
I20260925 02:23:19.727756 21630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:42609 every 8 connection(s)
I20260925 02:23:19.728076 21563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:23:19.728766 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:19.731549 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301: Bootstrap starting.
I20260925 02:23:19.731628 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21563
I20260925 02:23:19.731756 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:23:19.731957 11764 external_mini_cluster.cc:1428] Setting key 23d6c5b289bd4eb5ce5f6023f9993f8a
I20260925 02:23:19.732270 21631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:19.732659 21631 log.cc:824] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:19.733577 21631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301: No bootstrap required, opened a new log
I20260925 02:23:19.734714 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.732493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.11.125.62:42609)
0925 02:23:19.732706 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.732716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.732749 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:23:19.733363 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.733372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.733423 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.733588 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.734268 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.734272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.734273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.734277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.734324 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.734338 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.734434 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.734464 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.734520 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260925 02:23:19.735565 21631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } }
I20260925 02:23:19.735693 21631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:19.735715 21631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a590fee63f04d3a968f39e6d9dab301, State: Initialized, Role: FOLLOWER
I20260925 02:23:19.735826 21631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } }
I20260925 02:23:19.735898 21631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:23:19.735920 21631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:23:19.735947 21631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:19.736764 21631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } }
I20260925 02:23:19.736888 21631 leader_election.cc:302] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a590fee63f04d3a968f39e6d9dab301; no voters:
I20260925 02:23:19.737112 21631 leader_election.cc:288] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:23:19.737201 21636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:19.737376 21636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [term 1 LEADER]: Becoming Leader. State: Replica: 2a590fee63f04d3a968f39e6d9dab301, State: Running, Role: LEADER
I20260925 02:23:19.737406 21631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:23:19.737470 21636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } }
I20260925 02:23:19.737829 21637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a590fee63f04d3a968f39e6d9dab301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } } }
I20260925 02:23:19.737835 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a590fee63f04d3a968f39e6d9dab301. Latest consensus state: current_term: 1 leader_uuid: "2a590fee63f04d3a968f39e6d9dab301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a590fee63f04d3a968f39e6d9dab301" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 42609 } } }
I20260925 02:23:19.738044 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:19.737998 21637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:19.738476 21645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:23:19.738996 21645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:23:19.740471 21645 catalog_manager.cc:1347] Generated new cluster ID: 91893e1183cc4c9ebed3e6504e4da8a2
I20260925 02:23:19.740515 21645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:23:19.745049 21645 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:23:19.745210 21645 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:23:19.749338 21645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a590fee63f04d3a968f39e6d9dab301: Generated new TSK 0
I20260925 02:23:19.749524 21645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:23:19.755328 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--builtin_ntp_servers=127.11.125.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:23:19.837280 21655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.837507 21655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.837546 21655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.839006 21655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.839066 21655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.839110 21655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:23:19.840569 21655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.840832 21655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.841151 21655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.847091 21661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.847200 21655 server_base.cc:1034] running on GCE node
W20260925 02:23:19.847335 21660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.847101 21663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.847594 21655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.847838 21655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.848973 21655 hybrid_clock.cc:648] HybridClock initialized: now 1790302999848959 us; error 31 us; skew 500 ppm
I20260925 02:23:19.850303 21655 webserver.cc:460] Webserver started at http://127.11.125.1:46195/ using document root <none> and password file <none>
I20260925 02:23:19.850554 21655 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.850600 21655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.850745 21655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.851840 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "72426baa830b4f8ca8304cb30ca5bd41"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "4865c2f26e37aefe21af3fb0ac9d5917"
server_key_iv: "46a2afa8d50a5d755e3c7e5ab6ee1498"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.852216 21655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "72426baa830b4f8ca8304cb30ca5bd41"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "4865c2f26e37aefe21af3fb0ac9d5917"
server_key_iv: "46a2afa8d50a5d755e3c7e5ab6ee1498"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.853659 21655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:23:19.854471 21669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.854628 21655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:19.854741 21655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "72426baa830b4f8ca8304cb30ca5bd41"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "4865c2f26e37aefe21af3fb0ac9d5917"
server_key_iv: "46a2afa8d50a5d755e3c7e5ab6ee1498"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.854823 21655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.858260 21655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.858566 21655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.858708 21655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.858968 21655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:19.859349 21655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:19.859386 21655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.859407 21655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:19.859445 21655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.866765 21655 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:36199
I20260925 02:23:19.866811 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:36199 every 8 connection(s)
I20260925 02:23:19.867139 21655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:23:19.870362 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.867906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:49061 (local address 127.11.125.62:42609)
0925 02:23:19.867950 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.867952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.868165 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:23:19.868721 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.868723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.868730 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.868777 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.870090 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.870093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.870093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.870098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.870114 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.870118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.870166 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.870277 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.870292 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:23:19.870431 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21655
I20260925 02:23:19.870529 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:23:19.870607 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.867855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42609 (local address 127.11.125.1:49061)
0925 02:23:19.868039 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:19.868072 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:19.868598 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:19.868860 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:19.868864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:19.868873 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:19.869931 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:19.869932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:19.869960 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:19.869988 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:19.870176 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:19.870181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:19.870184 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:23:19.870287 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:23:19.870356 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260925 02:23:19.870730 11764 external_mini_cluster.cc:1428] Setting key 624fe8d8441d84d40b85159a86b7733d
I20260925 02:23:19.871117 21784 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42609
I20260925 02:23:19.871215 21784 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:19.871387 21784 heartbeater.cc:511] Master 127.11.125.62:42609 requested a full tablet report, sending...
I20260925 02:23:19.871807 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--builtin_ntp_servers=127.11.125.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:23:19.871958 21594 ts_manager.cc:194] Registered new tserver with Master: 72426baa830b4f8ca8304cb30ca5bd41 (127.11.125.1:36199)
W20260925 02:23:19.957726 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:19.958012 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:19.958047 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:19.959415 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:19.959461 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:19.959487 21787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:23:19.960785 21787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:19.961051 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:19.961308 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:19.966343 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:19.966315 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.966532 21787 server_base.cc:1034] running on GCE node
W20260925 02:23:19.966334 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:19.968678 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:19.969034 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:19.970374 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790302999970335 us; error 36 us; skew 500 ppm
I20260925 02:23:19.972272 21787 webserver.cc:460] Webserver started at http://127.11.125.2:43433/ using document root <none> and password file <none>
I20260925 02:23:19.972590 21787 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:19.972669 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:19.972826 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:19.974707 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "0bdad5807568487e934e75504951c87f"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "c6fcf99a7820f963da3b5ec8c4ce2b4c"
server_key_iv: "5fdc1b4905eef67fc89f5b693ed737cd"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.975149 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "0bdad5807568487e934e75504951c87f"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "c6fcf99a7820f963da3b5ec8c4ce2b4c"
server_key_iv: "5fdc1b4905eef67fc89f5b693ed737cd"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.976948 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:23:19.978128 21801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.978333 21787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:19.978437 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "0bdad5807568487e934e75504951c87f"
format_stamp: "Formatted at 2026-09-25 02:23:19 on dist-test-slave-hj7s"
server_key: "c6fcf99a7820f963da3b5ec8c4ce2b4c"
server_key_iv: "5fdc1b4905eef67fc89f5b693ed737cd"
server_key_version: "encryptionkey@0"
I20260925 02:23:19.978526 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:19.984450 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:19.984787 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:19.984977 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:19.985241 21787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:19.985601 21787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:19.985683 21787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.985719 21787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:19.985749 21787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:19.993785 21787 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:38977
I20260925 02:23:19.993871 21914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:38977 every 8 connection(s)
I20260925 02:23:19.994222 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:23:19.997128 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21787
I20260925 02:23:19.997260 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:23:19.997475 11764 external_mini_cluster.cc:1428] Setting key ecd6d3b0520ad349f01174e2eee40166
I20260925 02:23:19.997982 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.995005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:48351 (local address 127.11.125.62:42609)
0925 02:23:19.995047 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:19.995049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:19.995283 (+ 234us) server_negotiation.cc:407] Connection header received
0925 02:23:19.995937 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:19.995939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:19.995947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:19.996026 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:19.997623 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:19.997626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:19.997626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:19.997632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:19.997650 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:19.997656 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:19.997690 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:19.997836 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 02:23:19.997883 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:23:19.998147 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:19.994968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42609 (local address 127.11.125.2:48351)
0925 02:23:19.995159 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:19.995193 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:19.995783 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:19.996109 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:19.996113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:19.996123 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:19.997421 (+ 1298us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:19.997422 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:19.997463 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:19.997500 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:19.997733 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:19.997738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:19.997741 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:23:19.997841 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:23:19.997909 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260925 02:23:19.998693 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--builtin_ntp_servers=127.11.125.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:23:20.000313 21915 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42609
I20260925 02:23:20.000434 21915 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:20.000614 21915 heartbeater.cc:511] Master 127.11.125.62:42609 requested a full tablet report, sending...
I20260925 02:23:20.001012 21594 ts_manager.cc:194] Registered new tserver with Master: 0bdad5807568487e934e75504951c87f (127.11.125.2:38977)
W20260925 02:23:20.083067 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:20.083264 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:20.083288 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:20.084524 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:20.084611 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:20.084652 21918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:23:20.086117 21918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:20.086390 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:20.086688 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:20.091805 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:20.091754 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:20.091946 21918 server_base.cc:1034] running on GCE node
W20260925 02:23:20.091768 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:20.092305 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:20.092507 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:20.093647 21918 hybrid_clock.cc:648] HybridClock initialized: now 1790303000093634 us; error 31 us; skew 500 ppm
I20260925 02:23:20.094991 21918 webserver.cc:460] Webserver started at http://127.11.125.3:43629/ using document root <none> and password file <none>
I20260925 02:23:20.095290 21918 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:20.095353 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:20.095507 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:20.096592 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "6a2e8026b6764874887c0754a646edcc"
format_stamp: "Formatted at 2026-09-25 02:23:20 on dist-test-slave-hj7s"
server_key: "5afb4489bb75bbf009f54ee21a5430dd"
server_key_iv: "1821c90d6a48783b23be2c76e741c7f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:20.097009 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "6a2e8026b6764874887c0754a646edcc"
format_stamp: "Formatted at 2026-09-25 02:23:20 on dist-test-slave-hj7s"
server_key: "5afb4489bb75bbf009f54ee21a5430dd"
server_key_iv: "1821c90d6a48783b23be2c76e741c7f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:20.098327 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:20.099071 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:20.099272 21918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:20.099363 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "6a2e8026b6764874887c0754a646edcc"
format_stamp: "Formatted at 2026-09-25 02:23:20 on dist-test-slave-hj7s"
server_key: "5afb4489bb75bbf009f54ee21a5430dd"
server_key_iv: "1821c90d6a48783b23be2c76e741c7f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:20.099436 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:20.102819 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:20.103091 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:20.103225 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:20.103440 21918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:20.103749 21918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:20.103782 21918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:20.103832 21918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:20.103857 21918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:20.109697 21918 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:36237
I20260925 02:23:20.109753 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:36237 every 8 connection(s)
I20260925 02:23:20.110064 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:23:20.113302 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:20.110872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:42565 (local address 127.11.125.62:42609)
0925 02:23:20.110982 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:20.110985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:20.111348 (+ 363us) server_negotiation.cc:407] Connection header received
0925 02:23:20.111822 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:20.111823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:20.111830 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:20.111880 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:20.113130 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:20.113131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:20.113131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:23:20.113134 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:20.113146 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:23:20.113149 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:20.113193 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:20.113238 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:23:20.113250 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:23:20.113534 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:20.110983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:42609 (local address 127.11.125.3:42565)
0925 02:23:20.111208 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:20.111242 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:20.111714 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:20.111939 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:20.111942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:20.111950 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:20.112979 (+ 1029us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:23:20.112980 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:20.113018 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:23:20.113048 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:20.113191 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:20.113193 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:20.113196 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:23:20.113246 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:23:20.113296 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260925 02:23:20.114050 22048 heartbeater.cc:347] Connected to a master server at 127.11.125.62:42609
I20260925 02:23:20.114159 22048 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:20.114323 22048 heartbeater.cc:511] Master 127.11.125.62:42609 requested a full tablet report, sending...
I20260925 02:23:20.114701 21594 ts_manager.cc:194] Registered new tserver with Master: 6a2e8026b6764874887c0754a646edcc (127.11.125.3:36237)
I20260925 02:23:20.115631 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 21918
I20260925 02:23:20.115707 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:23:20.115908 11764 external_mini_cluster.cc:1428] Setting key 70d16ea3915f91da23df64c8307e1af7
I20260925 02:23:20.116676 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:23:20.118525 21634 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:23:20.118095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.11.125.62:42609)
0925 02:23:20.118134 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:20.118136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:20.118236 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:23:20.118303 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:20.118304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:20.118309 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:20.118355 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:20.118450 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41852: BlockingRecv error: recv got EOF from 127.0.0.1:41852 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:23:20.118618 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21655
I20260925 02:23:20.124931 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21787
I20260925 02:23:20.129834 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21918
I20260925 02:23:20.134878 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 21563
2026-09-25T02:23:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:23:20 dist-test-slave-hj7s krb5kdc[22058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:23:20 dist-test-slave-hj7s krb5kdc[22058](info): set up 2 sockets
Sep 25 02:23:20 dist-test-slave-hj7s krb5kdc[22058](info): commencing operation
krb5kdc: starting...
W20260925 02:23:21.204114 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:23:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:23:21.324877 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33749
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:21.408682 22071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:21.408973 22071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:21.409029 22071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:21.410210 22071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:23:21.410279 22071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:21.410303 22071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:21.410334 22071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:23:21.410388 22071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:23:21.411916 22071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:33749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:21.412197 22071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:21.412431 22071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:21.417553 22079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.417553 22077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.417699 22071 server_base.cc:1034] running on GCE node
W20260925 02:23:21.417805 22076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.418089 22071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:21.418352 22071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:21.419526 22071 hybrid_clock.cc:648] HybridClock initialized: now 1790303001419501 us; error 37 us; skew 500 ppm
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:21.422850 22071 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:23:21.423255 22071 webserver.cc:460] Webserver started at http://127.11.125.62:33301/ using document root <none> and password file <none>
I20260925 02:23:21.423497 22071 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:21.423544 22071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:21.423688 22071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:21.424813 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "b90542d5842a463a8f14fcbe9700dc78"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "43160f2d2879efe4b0a3ebda370b5b3a"
server_key_iv: "4da6db1b76a77a4ea3830db0b07500f6"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.425264 22071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "b90542d5842a463a8f14fcbe9700dc78"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "43160f2d2879efe4b0a3ebda370b5b3a"
server_key_iv: "4da6db1b76a77a4ea3830db0b07500f6"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.426654 22071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:21.427541 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.427851 22071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:21.427929 22071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "b90542d5842a463a8f14fcbe9700dc78"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "43160f2d2879efe4b0a3ebda370b5b3a"
server_key_iv: "4da6db1b76a77a4ea3830db0b07500f6"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.428023 22071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:21.449834 22071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:21.452219 22071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:21.452389 22071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:21.457098 22071 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:33749
I20260925 02:23:21.457154 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:33749 every 8 connection(s)
I20260925 02:23:21.457484 22071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:23:21.458487 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:21.460132 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 22071
I20260925 02:23:21.460246 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:23:21.460464 11764 external_mini_cluster.cc:1428] Setting key 693c25070253c5ce9a89c1f01d217110
I20260925 02:23:21.461169 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: Bootstrap starting.
I20260925 02:23:21.461782 22140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:21.462110 22140 log.cc:824] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:21.462822 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: No bootstrap required, opened a new log
I20260925 02:23:21.464682 22140 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } }
I20260925 02:23:21.464867 22140 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:21.464955 22140 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90542d5842a463a8f14fcbe9700dc78, State: Initialized, Role: FOLLOWER
I20260925 02:23:21.465130 22140 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } }
I20260925 02:23:21.465233 22140 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:23:21.465281 22140 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:23:21.465350 22140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:21.466178 22140 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } }
I20260925 02:23:21.466311 22140 leader_election.cc:302] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90542d5842a463a8f14fcbe9700dc78; no voters:
I20260925 02:23:21.466467 22140 leader_election.cc:288] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:23:21.466543 22145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:21.466742 22140 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:23:21.466763 22145 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [term 1 LEADER]: Becoming Leader. State: Replica: b90542d5842a463a8f14fcbe9700dc78, State: Running, Role: LEADER
I20260925 02:23:21.466964 22145 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } }
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:21.467341 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90542d5842a463a8f14fcbe9700dc78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } } }
I20260925 02:23:21.467355 22147 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90542d5842a463a8f14fcbe9700dc78. Latest consensus state: current_term: 1 leader_uuid: "b90542d5842a463a8f14fcbe9700dc78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90542d5842a463a8f14fcbe9700dc78" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 33749 } } }
I20260925 02:23:21.467450 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:21.467454 22147 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78 [sys.catalog]: This master's current role is: LEADER
I20260925 02:23:21.467730 22154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:23:21.468289 22154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:23:21.470032 22154 catalog_manager.cc:1347] Generated new cluster ID: cb616446e10943938f42b29937bfb1e7
I20260925 02:23:21.470101 22154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:23:21.473234 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:21.461054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.11.125.62:33749)
0925 02:23:21.461241 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:21.461251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:21.461310 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:23:21.461744 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:21.461750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:21.461798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:21.461956 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:21.462636 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.463583 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.464350 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.464753 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.464878 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:21.468154 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:21.468159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:21.468159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:21.468174 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:21.471353 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.471776 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.471780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.471780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.471890 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.472108 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.472109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.472110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.472426 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:21.472565 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:21.472847 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:23:21.473016 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260925 02:23:21.490932 22154 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:23:21.491492 22154 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:23:21.497262 22154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: Generated new TSK 0
I20260925 02:23:21.497493 22154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:23:21.527967 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:21.611783 22166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:21.612007 22166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:21.612031 22166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:21.613302 22166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:21.613348 22166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:21.613376 22166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:23:21.614935 22166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:21.615207 22166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:21.615454 22166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:21.620822 22172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.620780 22171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.620837 22166 server_base.cc:1034] running on GCE node
W20260925 02:23:21.620780 22174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.621276 22166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:21.621592 22166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:21.622761 22166 hybrid_clock.cc:648] HybridClock initialized: now 1790303001622746 us; error 33 us; skew 500 ppm
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:21.625367 22166 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:23:21.625859 22166 webserver.cc:460] Webserver started at http://127.11.125.1:34905/ using document root <none> and password file <none>
I20260925 02:23:21.626101 22166 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:21.626168 22166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:21.626313 22166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:21.627388 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "11d7fda65bd641d8adb034ac47eb4ad6"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5b154648651fc90192128c3f19c5427c"
server_key_iv: "0cf657469bc10f2232c618b1437a6212"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.627763 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "11d7fda65bd641d8adb034ac47eb4ad6"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5b154648651fc90192128c3f19c5427c"
server_key_iv: "0cf657469bc10f2232c618b1437a6212"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.629164 22166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:23:21.629933 22181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.630112 22166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:21.630215 22166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "11d7fda65bd641d8adb034ac47eb4ad6"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5b154648651fc90192128c3f19c5427c"
server_key_iv: "0cf657469bc10f2232c618b1437a6212"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.630296 22166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:21.640302 22166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:21.642315 22166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:21.642477 22166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:21.642716 22166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:21.643040 22166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:21.643103 22166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.643146 22166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:21.643174 22166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.649223 22166 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:36895
I20260925 02:23:21.649267 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:36895 every 8 connection(s)
I20260925 02:23:21.649585 22166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:23:21.653129 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 22166
I20260925 02:23:21.653208 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:23:21.653395 11764 external_mini_cluster.cc:1428] Setting key 713f6c624f35e32bb838a61533ef6856
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:21.659741 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:21.650301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:60483 (local address 127.11.125.62:33749)
0925 02:23:21.650368 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:21.650371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:21.650605 (+ 234us) server_negotiation.cc:407] Connection header received
0925 02:23:21.651087 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:21.651088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:21.651098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:21.651152 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:21.652126 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.652652 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.653919 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.654220 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.654298 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:21.656709 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:21.656711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:21.656712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:21.656719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:21.658229 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.658702 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.658704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.658704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.658770 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.659045 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.659046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.659046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.659172 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:21.659251 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:21.659546 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 02:23:21.659646 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:23:21.659958 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:21.650288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33749 (local address 127.11.125.1:60483)
0925 02:23:21.650487 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:21.650524 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:21.651004 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:21.651205 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:21.651209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:21.651815 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:21.652027 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:21.652033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.652735 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.652736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:21.653834 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:21.653837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.654330 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.654331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:21.654442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:21.655035 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:21.655044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:21.656597 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:21.658349 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:21.658359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:21.658361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:21.658608 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:21.658844 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:21.658845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:21.658845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:21.658969 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:21.659242 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:21.659250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:21.659403 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 02:23:21.659512 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:23:21.659689 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260925 02:23:21.660738 22296 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33749
I20260925 02:23:21.660832 22296 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:21.661072 22296 heartbeater.cc:511] Master 127.11.125.62:33749 requested a full tablet report, sending...
I20260925 02:23:21.661665 22103 ts_manager.cc:194] Registered new tserver with Master: 11d7fda65bd641d8adb034ac47eb4ad6 (127.11.125.1:36895)
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:23:21.662367 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:60483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:23:21.680513 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:21.764221 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:21.764436 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:21.764460 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:21.765758 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:21.765805 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:21.765874 22301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:23:21.767570 22301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:21.767825 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:21.768080 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:21.773296 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.773350 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.773296 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.773550 22301 server_base.cc:1034] running on GCE node
I20260925 02:23:21.773782 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:21.774035 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:21.775202 22301 hybrid_clock.cc:648] HybridClock initialized: now 1790303001775168 us; error 43 us; skew 500 ppm
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:21.777707 22301 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:23:21.778199 22301 webserver.cc:460] Webserver started at http://127.11.125.2:37341/ using document root <none> and password file <none>
I20260925 02:23:21.778434 22301 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:21.778508 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:21.778672 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:21.779748 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "14d916d0e76a401ba11740235f9107d9"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "1dc84e75d9ab287613fc54e0c6b1893f"
server_key_iv: "c3e844d681332fe16d3923f3a651a28a"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.780128 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "14d916d0e76a401ba11740235f9107d9"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "1dc84e75d9ab287613fc54e0c6b1893f"
server_key_iv: "c3e844d681332fe16d3923f3a651a28a"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.781670 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:23:21.782478 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.782677 22301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:23:21.782791 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "14d916d0e76a401ba11740235f9107d9"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "1dc84e75d9ab287613fc54e0c6b1893f"
server_key_iv: "c3e844d681332fe16d3923f3a651a28a"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.782882 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:21.823860 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:21.825896 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:21.826066 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:21.826327 22301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:21.826678 22301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:21.826715 22301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.826761 22301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:21.826788 22301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.833184 22301 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:44229
I20260925 02:23:21.833279 22430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:44229 every 8 connection(s)
I20260925 02:23:21.833638 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:23:21.836021 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 22301
I20260925 02:23:21.836134 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:23:21.836396 11764 external_mini_cluster.cc:1428] Setting key 37e2645ff381025c39d67ecaec9ba315
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:21.845533 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:21.834503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:52491 (local address 127.11.125.62:33749)
0925 02:23:21.834574 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:21.834576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:21.834838 (+ 262us) server_negotiation.cc:407] Connection header received
0925 02:23:21.835344 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:21.835345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:21.835356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:21.835412 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:21.836709 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.837448 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.839012 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.839347 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.839428 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:21.841992 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:21.841993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:21.841994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:21.842004 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:21.843834 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.844264 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.844265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.844266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.844340 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:21.844607 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:21.844609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:21.844609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:21.844757 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:21.844845 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:21.845300 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 02:23:21.845394 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:23:21.845777 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:21.834504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33749 (local address 127.11.125.2:52491)
0925 02:23:21.834673 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:21.834716 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:21.835249 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:21.835478 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:21.835481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:21.836254 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:21.836588 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:21.836596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.837556 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.837557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:21.838923 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:21.838926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:21.839439 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:21.839440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:21.839540 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:21.840098 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:21.840107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:21.841882 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:21.843935 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:21.843947 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:21.843949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:21.844180 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:21.844414 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:21.844415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:21.844415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:21.844514 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:21.844868 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:21.844875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:21.845119 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 02:23:21.845246 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:23:21.845415 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:23:21.846416 22431 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33749
I20260925 02:23:21.846541 22431 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:21.846807 22431 heartbeater.cc:511] Master 127.11.125.62:33749 requested a full tablet report, sending...
I20260925 02:23:21.847268 22103 ts_manager.cc:194] Registered new tserver with Master: 14d916d0e76a401ba11740235f9107d9 (127.11.125.2:44229)
I20260925 02:23:21.847697 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:52491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:23:21.864460 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:23:21.949244 22436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:23:21.949501 22436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:23:21.949536 22436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:23:21.950745 22436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:23:21.950834 22436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:23:21.950884 22436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:23:21.952433 22436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:33749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:23:21.952723 22436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:23:21.952997 22436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:23:21.958197 22441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.958209 22442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:23:21.958192 22444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:23:21.958618 22436 server_base.cc:1034] running on GCE node
I20260925 02:23:21.958843 22436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:23:21.959172 22436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:23:21.960354 22436 hybrid_clock.cc:648] HybridClock initialized: now 1790303001960347 us; error 36 us; skew 500 ppm
Sep 25 02:23:21 dist-test-slave-hj7s krb5kdc[22058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:23:21.963192 22436 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:23:21.963708 22436 webserver.cc:460] Webserver started at http://127.11.125.3:38635/ using document root <none> and password file <none>
I20260925 02:23:21.963977 22436 fs_manager.cc:359] Metadata directory not provided
I20260925 02:23:21.964049 22436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:23:21.964231 22436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:23:21.965857 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "d181c8ee53234ff49dfe4339f1342f4d"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5fa39fe31e86b8562e7ebed13fc0a274"
server_key_iv: "5d964ea07f609beae034bde8bcd4a6f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.966275 22436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "d181c8ee53234ff49dfe4339f1342f4d"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5fa39fe31e86b8562e7ebed13fc0a274"
server_key_iv: "5d964ea07f609beae034bde8bcd4a6f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.968026 22436 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:23:21.968799 22451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.969053 22436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:23:21.969159 22436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "d181c8ee53234ff49dfe4339f1342f4d"
format_stamp: "Formatted at 2026-09-25 02:23:21 on dist-test-slave-hj7s"
server_key: "5fa39fe31e86b8562e7ebed13fc0a274"
server_key_iv: "5d964ea07f609beae034bde8bcd4a6f7"
server_key_version: "encryptionkey@0"
I20260925 02:23:21.969254 22436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:23:21.994024 22436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:23:21.996012 22436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:23:21.996188 22436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:23:21.996443 22436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:23:21.996766 22436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:23:21.996802 22436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:21.996882 22436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:23:21.996968 22436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:23:22.003132 22436 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:42181
I20260925 02:23:22.003180 22565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:42181 every 8 connection(s)
I20260925 02:23:22.003502 22436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:23:22 dist-test-slave-hj7s krb5kdc[22058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:23:22.009717 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 22436
I20260925 02:23:22.009805 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:23:22.010003 11764 external_mini_cluster.cc:1428] Setting key 7589b5c934ac927c045494fb15ea885e
I20260925 02:23:22.013111 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.004186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:37203 (local address 127.11.125.62:33749)
0925 02:23:22.004257 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.004259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.004483 (+ 224us) server_negotiation.cc:407] Connection header received
0925 02:23:22.005037 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.005038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.005049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.005102 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:22.006044 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.006557 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.007558 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.007751 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.007808 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.010141 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:22.010143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:22.010143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:22.010159 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:22.011653 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:22.012101 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:22.012102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:22.012103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:22.012166 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:22.012427 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:22.012428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:22.012428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:22.012547 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:22.012627 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.012929 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.013011 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:23:22.013321 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.004185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:33749 (local address 127.11.125.3:37203)
0925 02:23:22.004354 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.004391 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.004888 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.005189 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.005193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.005757 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:23:22.005963 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.005968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.006642 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.006643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.007480 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.007482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.007833 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.007834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.007941 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.008484 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:23:22.008494 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:23:22.010048 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:23:22.011780 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:22.011787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:22.011795 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:22.012020 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:22.012238 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:23:22.012239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:23:22.012239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:23:22.012356 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:23:22.012618 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:23:22.012625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:23:22.012768 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.012884 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.013072 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260925 02:23:22.013777 22566 heartbeater.cc:347] Connected to a master server at 127.11.125.62:33749
I20260925 02:23:22.013898 22566 heartbeater.cc:464] Registering TS with master...
I20260925 02:23:22.014075 22566 heartbeater.cc:511] Master 127.11.125.62:33749 requested a full tablet report, sending...
I20260925 02:23:22.014462 22103 ts_manager.cc:194] Registered new tserver with Master: d181c8ee53234ff49dfe4339f1342f4d (127.11.125.3:42181)
I20260925 02:23:22.014844 22103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:37203
I20260925 02:23:22.021646 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:23:22.028352 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.022928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.11.125.62:33749)
0925 02:23:22.022996 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.022997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.023078 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:23:22.023118 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.023119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.023128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.023177 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:23:22.023699 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.024191 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.024870 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.025104 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.025156 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.026218 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:23:22.026219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:23:22.026219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:23:22.026225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:23:22.027414 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:22.027723 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:22.027724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:22.027724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:22.027774 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:23:22.027940 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:23:22.027941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:23:22.027941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:23:22.028076 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:23:22.028153 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.028223 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.028283 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:23:22.030123 22103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:23:22.031917 22103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:23:22.038861 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:36895 (local address 127.0.0.1:37532)
0925 02:23:22.035132 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.035142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.035179 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.035495 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.035496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.035499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:23:22.035550 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.035551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036442 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.036443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.037795 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.037797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038552 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.038633 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.038647 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.038720 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.038779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260925 02:23:22.038873 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.11.125.1:36895)
0925 02:23:22.035256 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.035261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.035272 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:23:22.035313 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.035315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.035331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.035407 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:23:22.035640 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036341 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.037889 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038456 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038549 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.038614 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.038748 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.038800 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":85,"threads_started":1}
I20260925 02:23:22.039170 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.11.125.3:42181)
0925 02:23:22.034940 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.034945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.035065 (+ 120us) server_negotiation.cc:407] Connection header received
0925 02:23:22.035111 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.035113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.035128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.035200 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:23:22.035474 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036631 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.037937 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038717 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038791 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.038847 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.039046 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.039097 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260925 02:23:22.039335 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.11.125.2:44229)
0925 02:23:22.035017 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.035020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.035048 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:23:22.035092 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.035095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.035112 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.035198 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:23:22.035534 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036632 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.037993 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038717 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038791 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.038847 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.039201 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.039255 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":92,"threads_started":1}
I20260925 02:23:22.039155 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:42181 (local address 127.0.0.1:60412)
0925 02:23:22.034974 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.034985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.035053 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.035297 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.035301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.035303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:23:22.035387 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.035390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036747 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.036748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.037850 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.037852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038885 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.038958 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.038963 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.039021 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.039069 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":14,"thread_start_us":102,"threads_started":1}
I20260925 02:23:22.039460 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.034544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:44229 (local address 127.0.0.1:49756)
0925 02:23:22.034659 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.034699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.034768 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.035312 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.035313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.035315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:23:22.035387 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.035389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.036981 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.036983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.037903 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.037906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.038888 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.038889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.039012 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.039016 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.039061 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.039092 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260925 02:23:22.040138 22365 tablet_service.cc:1467] Processing CreateTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 (DEFAULT_TABLE table=test-table [id=5236db45500d4a20b1335b6b9b3396e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:22.040129 22500 tablet_service.cc:1467] Processing CreateTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 (DEFAULT_TABLE table=test-table [id=5236db45500d4a20b1335b6b9b3396e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:22.040143 22230 tablet_service.cc:1467] Processing CreateTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 (DEFAULT_TABLE table=test-table [id=5236db45500d4a20b1335b6b9b3396e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:23:22.040495 22365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57803aaa58b413ea54c0558f3b5f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:22.040501 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57803aaa58b413ea54c0558f3b5f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:22.040501 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57803aaa58b413ea54c0558f3b5f3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:23:22.043310 22583 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Bootstrap starting.
I20260925 02:23:22.044049 22583 tablet_bootstrap.cc:654] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:22.044057 22585 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Bootstrap starting.
I20260925 02:23:22.044098 22584 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Bootstrap starting.
I20260925 02:23:22.044668 22584 tablet_bootstrap.cc:654] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:22.044737 22583 log.cc:824] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:22.045328 22584 log.cc:824] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:22.045616 22585 tablet_bootstrap.cc:654] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Neither blocks nor log segments found. Creating new log.
I20260925 02:23:22.046096 22583 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: No bootstrap required, opened a new log
I20260925 02:23:22.046175 22583 ts_tablet_manager.cc:1397] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 02:23:22.046185 22585 log.cc:824] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:23:22.046928 22585 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: No bootstrap required, opened a new log
I20260925 02:23:22.047008 22585 ts_tablet_manager.cc:1397] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:23:22.048235 22583 raft_consensus.cc:348] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.048418 22583 raft_consensus.cc:374] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:22.048465 22583 raft_consensus.cc:729] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d916d0e76a401ba11740235f9107d9, State: Initialized, Role: FOLLOWER
I20260925 02:23:22.048591 22583 consensus_queue.cc:260] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.048934 22583 ts_tablet_manager.cc:1428] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:23:22.048986 22431 heartbeater.cc:503] Master 127.11.125.62:33749 was elected leader, sending a full tablet report...
I20260925 02:23:22.049045 22585 raft_consensus.cc:348] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.049190 22585 raft_consensus.cc:374] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:22.049213 22585 raft_consensus.cc:729] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d7fda65bd641d8adb034ac47eb4ad6, State: Initialized, Role: FOLLOWER
I20260925 02:23:22.049245 22584 tablet_bootstrap.cc:492] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: No bootstrap required, opened a new log
I20260925 02:23:22.049365 22585 consensus_queue.cc:260] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.049422 22584 ts_tablet_manager.cc:1397] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 02:23:22.049665 22296 heartbeater.cc:503] Master 127.11.125.62:33749 was elected leader, sending a full tablet report...
I20260925 02:23:22.049659 22585 ts_tablet_manager.cc:1428] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:23:22.051259 22584 raft_consensus.cc:348] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.051431 22584 raft_consensus.cc:374] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:23:22.051479 22584 raft_consensus.cc:729] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d181c8ee53234ff49dfe4339f1342f4d, State: Initialized, Role: FOLLOWER
I20260925 02:23:22.051590 22584 consensus_queue.cc:260] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.051856 22584 ts_tablet_manager.cc:1428] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 02:23:22.051913 22566 heartbeater.cc:503] Master 127.11.125.62:33749 was elected leader, sending a full tablet report...
W20260925 02:23:22.084270 22432 tablet.cc:2369] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:23:22.150357 22297 tablet.cc:2369] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:23:22.163853 22590 raft_consensus.cc:482] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:23:22.164031 22590 raft_consensus.cc:504] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.164431 22590 leader_election.cc:288] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d916d0e76a401ba11740235f9107d9 (127.11.125.2:44229), d181c8ee53234ff49dfe4339f1342f4d (127.11.125.3:42181)
I20260925 02:23:22.167662 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.164670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50291 (local address 127.11.125.2:44229)
0925 02:23:22.164740 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.164742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.164882 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:23:22.164990 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.164991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.165000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.165069 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:23:22.165257 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.165736 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.166794 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.167298 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.167377 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.167387 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.167549 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.167586 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:23:22.167670 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.164731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50341 (local address 127.11.125.3:42181)
0925 02:23:22.164784 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.164786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.164917 (+ 131us) server_negotiation.cc:407] Connection header received
0925 02:23:22.164990 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.164991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.165001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.165091 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:23:22.165258 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.165736 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.166833 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.167299 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.167383 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.167392 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.167558 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.167600 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:23:22.167695 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.164609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:42181 (local address 127.11.125.1:50341)
0925 02:23:22.164804 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.164815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.164912 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.165095 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.165096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.165098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:23:22.165185 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.165186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.165832 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.165833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.166672 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.166674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.167383 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.167384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.167472 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.167476 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.167538 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.167584 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260925 02:23:22.167706 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.164607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:44229 (local address 127.11.125.1:50291)
0925 02:23:22.164810 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.164819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:23:22.164900 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:23:22.165090 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:23:22.165091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:23:22.165093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:23:22.165180 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.165181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.165825 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.165826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.166686 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:23:22.166688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.167388 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.167389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:23:22.167464 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.167468 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:23:22.167531 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:23:22.167581 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260925 02:23:22.168097 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57803aaa58b413ea54c0558f3b5f3a1" candidate_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d181c8ee53234ff49dfe4339f1342f4d" is_pre_election: true
I20260925 02:23:22.168294 22520 raft_consensus.cc:2454] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d7fda65bd641d8adb034ac47eb4ad6 in term 0.
I20260925 02:23:22.168455 22385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57803aaa58b413ea54c0558f3b5f3a1" candidate_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d916d0e76a401ba11740235f9107d9" is_pre_election: true
I20260925 02:23:22.168529 22184 leader_election.cc:302] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d7fda65bd641d8adb034ac47eb4ad6, d181c8ee53234ff49dfe4339f1342f4d; no voters:
I20260925 02:23:22.168589 22385 raft_consensus.cc:2454] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11d7fda65bd641d8adb034ac47eb4ad6 in term 0.
I20260925 02:23:22.168697 22590 raft_consensus.cc:2781] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:23:22.168759 22590 raft_consensus.cc:482] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:23:22.168787 22590 raft_consensus.cc:3037] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:22.169546 22590 raft_consensus.cc:504] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.169701 22590 leader_election.cc:288] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [CANDIDATE]: Term 1 election: Requested vote from peers 14d916d0e76a401ba11740235f9107d9 (127.11.125.2:44229), d181c8ee53234ff49dfe4339f1342f4d (127.11.125.3:42181)
I20260925 02:23:22.169914 22385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57803aaa58b413ea54c0558f3b5f3a1" candidate_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d916d0e76a401ba11740235f9107d9"
I20260925 02:23:22.170007 22385 raft_consensus.cc:3037] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:22.170740 22385 raft_consensus.cc:2454] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d7fda65bd641d8adb034ac47eb4ad6 in term 1.
I20260925 02:23:22.170902 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57803aaa58b413ea54c0558f3b5f3a1" candidate_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d181c8ee53234ff49dfe4339f1342f4d"
I20260925 02:23:22.170943 22185 leader_election.cc:302] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d7fda65bd641d8adb034ac47eb4ad6, 14d916d0e76a401ba11740235f9107d9; no voters:
I20260925 02:23:22.170991 22520 raft_consensus.cc:3037] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:23:22.171053 22590 raft_consensus.cc:2781] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:23:22.171224 22590 raft_consensus.cc:686] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 11d7fda65bd641d8adb034ac47eb4ad6, State: Running, Role: LEADER
I20260925 02:23:22.171342 22590 consensus_queue.cc:237] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } }
I20260925 02:23:22.172036 22520 raft_consensus.cc:2454] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11d7fda65bd641d8adb034ac47eb4ad6 in term 1.
I20260925 02:23:22.172638 22102 catalog_manager.cc:5495] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11d7fda65bd641d8adb034ac47eb4ad6 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d7fda65bd641d8adb034ac47eb4ad6" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 36895 } health_report { overall_health: HEALTHY } } }
I20260925 02:23:22.183391 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.181233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.11.125.1:36895)
0925 02:23:22.181304 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:23:22.181306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:23:22.181314 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:23:22.181345 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:23:22.181345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:23:22.181362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:23:22.181422 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:23:22.181536 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.182011 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.182734 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:23:22.182924 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:23:22.182974 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:23:22.183098 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:23:22.183194 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:23:22.183249 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:23:22.183278 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 02:23:22.183294 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:23:22.185230 22520 raft_consensus.cc:1261] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 1 FOLLOWER]: Refusing update from remote peer 11d7fda65bd641d8adb034ac47eb4ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:22.185228 22385 raft_consensus.cc:1261] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 1 FOLLOWER]: Refusing update from remote peer 11d7fda65bd641d8adb034ac47eb4ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:23:22.185658 22590 consensus_queue.cc:1035] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d181c8ee53234ff49dfe4339f1342f4d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:22.185827 22594 consensus_queue.cc:1035] T f57803aaa58b413ea54c0558f3b5f3a1 P 11d7fda65bd641d8adb034ac47eb4ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d916d0e76a401ba11740235f9107d9" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 44229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:23:22.188046 22597 mvcc.cc:205] Tried to move back new op lower bound from 7333081096947875840 to 7333081096894566400. Current Snapshot: MvccSnapshot[applied={T|T < 7333081096947875840}]
I20260925 02:23:22.188714 22599 mvcc.cc:205] Tried to move back new op lower bound from 7333081096947875840 to 7333081096894566400. Current Snapshot: MvccSnapshot[applied={T|T < 7333081096947875840}]
I20260925 02:23:22.188822 22598 mvcc.cc:205] Tried to move back new op lower bound from 7333081096947875840 to 7333081096894566400. Current Snapshot: MvccSnapshot[applied={T|T < 7333081096947875840}]
I20260925 02:23:22.190538 22102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:22.190647 22102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:23:22.191387 22102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90542d5842a463a8f14fcbe9700dc78: Sending DeleteTablet for 3 replicas of tablet f57803aaa58b413ea54c0558f3b5f3a1
I20260925 02:23:22.191762 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37532
I20260925 02:23:22.191769 22500 tablet_service.cc:1514] Processing DeleteTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:60412
I20260925 02:23:22.191781 22365 tablet_service.cc:1514] Processing DeleteTablet for tablet f57803aaa58b413ea54c0558f3b5f3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:23:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49756
I20260925 02:23:22.192097 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 22166
I20260925 02:23:22.192835 22610 tablet_replica.cc:330] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: stopping tablet replica
I20260925 02:23:22.193007 22610 raft_consensus.cc:2229] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:22.192162 22607 tablet_replica.cc:330] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: stopping tablet replica
I20260925 02:23:22.193143 22610 raft_consensus.cc:2258] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:22.193161 22607 raft_consensus.cc:2229] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:23:22.193358 22607 raft_consensus.cc:2258] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:23:22.193562 22610 ts_tablet_manager.cc:1905] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:22.193804 22607 ts_tablet_manager.cc:1905] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:23:22.195178 22610 ts_tablet_manager.cc:1918] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:23:22.195269 22610 log.cc:1196] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/f57803aaa58b413ea54c0558f3b5f3a1
I20260925 02:23:22.195585 22610 ts_tablet_manager.cc:1939] T f57803aaa58b413ea54c0558f3b5f3a1 P d181c8ee53234ff49dfe4339f1342f4d: Deleting consensus metadata
I20260925 02:23:22.195734 22607 ts_tablet_manager.cc:1918] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:23:22.195832 22607 log.cc:1196] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/f57803aaa58b413ea54c0558f3b5f3a1
I20260925 02:23:22.196009 22089 catalog_manager.cc:4841] TS d181c8ee53234ff49dfe4339f1342f4d (127.11.125.3:42181): tablet f57803aaa58b413ea54c0558f3b5f3a1 (table test-table [id=5236db45500d4a20b1335b6b9b3396e3]) successfully deleted
I20260925 02:23:22.196113 22607 ts_tablet_manager.cc:1939] T f57803aaa58b413ea54c0558f3b5f3a1 P 14d916d0e76a401ba11740235f9107d9: Deleting consensus metadata
I20260925 02:23:22.196494 22090 catalog_manager.cc:4841] TS 14d916d0e76a401ba11740235f9107d9 (127.11.125.2:44229): tablet f57803aaa58b413ea54c0558f3b5f3a1 (table test-table [id=5236db45500d4a20b1335b6b9b3396e3]) successfully deleted
W20260925 02:23:22.199591 22089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:36895 (error 108)
I20260925 02:23:22.199764 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 22301
I20260925 02:23:22.200708 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:23:22.200571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:36895 (local address 127.0.0.1:37548)
0925 02:23:22.200623 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:23:22.200665 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:36895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:23:22.200837 22089 catalog_manager.cc:4584] TS 11d7fda65bd641d8adb034ac47eb4ad6 (127.11.125.1:36895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f57803aaa58b413ea54c0558f3b5f3a1: Network error: Client connection negotiation failed: client connection to 127.11.125.1:36895: connect: Connection refused (error 111)
I20260925 02:23:22.205185 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 22436
I20260925 02:23:22.210973 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 22071
2026-09-25T02:23:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2076 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:23:22 dist-test-slave-hj7s krb5kdc[22614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:23:22 dist-test-slave-hj7s krb5kdc[22614](info): set up 2 sockets
Sep 25 02:23:22 dist-test-slave-hj7s krb5kdc[22614](info): commencing operation
krb5kdc: starting...
W20260925 02:23:23.259896 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:23:23 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:23:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangeradmin_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangeradmin_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangerlookup_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangerlookup_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
I20260925 02:23:23.422075 11764 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task70c4Wq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/postgres -l logfile start
2026-09-25 02:23:26.043 UTC [22644] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:23:26.043 UTC [22644] LOG: listening on IPv4 address "127.11.125.20", port 48985
2026-09-25 02:23:26.047 UTC [22644] LOG: listening on Unix socket "/tmp/.s.PGSQL.48985"
2026-09-25 02:23:26.068 UTC [22647] LOG: database system was shut down at 2026-09-25 02:23:24 UTC
2026-09-25 02:23:26.071 UTC [22644] LOG: database system is ready to accept connections
I20260925 02:23:27.096148 11764 mini_postgres.cc:96] Postgres bound to 48985
2026-09-25 02:23:27.104 UTC [22655] FATAL: database "slave" does not exist
127.11.125.20:48985 - accepting connections
I20260925 02:23:27.105517 11764 mini_ranger.cc:161] Starting Ranger...
I20260925 02:23:27.123138 11764 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:23:27.617651 11764 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:23:27.617720 11764 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin
2026-09-25 02:23:27,909 [I] DB FLAVOR :POSTGRES
2026-09-25 02:23:27,910 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:23:27,910 [I] Checking connection..
2026-09-25 02:23:27,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:23:28,166 [I] Checking connection passed.
2026-09-25 02:23:28,167 [I] --------- Verifying version history table ---------
2026-09-25 02:23:28,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:23:28,402 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:23:28,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:23:28,654 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:23:28,655 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:23:28,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:23:28.885 UTC [22719] WARNING: there is no transaction in progress
2026-09-25 02:23:28,891 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:23:28,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:23:29,141 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:23:29,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:23:29,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:23:29,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:23:29,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hj7s.c.gcp-upstream.internal','N') ;"
2026-09-25 02:23:29,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:23:30,149 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:23:30,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:23:30,388 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:23:30,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:23:30,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:23:30,895 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:23:30,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:23:31.879 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:31.896 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:31.913 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:31.928 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:31.945 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:31.963 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.028 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.035 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.049 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.059 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.069 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.084 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.094 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.101 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.108 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32.115 UTC [22827] WARNING: there is no transaction in progress
2026-09-25 02:23:32,785 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:23:32,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:23:33,066 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:23:33,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:23:33,305 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:23:33,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:23:33,562 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:23:33,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:23:33,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:23:34,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hj7s.c.gcp-upstream.internal';"
2026-09-25 02:23:34,314 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:23:34,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:23:34,584 [I] DB_PATCHES have already been applied
I20260925 02:23:34.589919 11764 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task70c4Wq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:23:34.589986 11764 mini_ranger.cc:193] Using host: 127.11.125.20
I20260925 02:23:34.593164 11764 mini_ranger.cc:227] Ranger admin URL: http://127.11.125.20:33429
Sep 25, 2026 2:23:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:23:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:23:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:23:35.714092 11764 mini_ranger.cc:160] Time spent starting Ranger: real 8.609s user 0.003s sys 0.006s
Sep 25, 2026 2:23:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:23:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:23:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:23:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:23:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:23:56.820058 11764 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangerkms_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/rangerkms_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.125.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.125.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:23:56 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303036, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:23:56.906682 11764 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:23:56.915772 11764 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:23:57.833748 11764 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:23:57.833849 11764 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms
I20260925 02:23:57.833873 11764 mini_ranger_kms.cc:227] Using postgres at 127.11.125.20:48985
2026-09-25 02:23:58,038 [I] DB FLAVOR :POSTGRES
2026-09-25 02:23:58,038 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:23:58,038 [I] Checking connection
2026-09-25 02:23:58,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:23:58,286 [I] connection success
2026-09-25 02:23:58,286 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:23:58,286 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:23:58,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:23:58,522 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:23:58,522 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:23:58,522 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:23:58,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:48985/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:23:58,791 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:23:59.261927 11764 mini_ranger_kms.cc:310] Created kms service
I20260925 02:23:59.430181 11764 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:23:59.476343 11764 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:23:59 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303036, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:23:59.864346 11764 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:23:59.864463 11764 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms:/tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:23:59.864485 11764 mini_ranger_kms.cc:240] Using host: 127.11.125.20
I20260925 02:23:59.867991 11764 mini_ranger_kms.cc:276] Ranger KMS PID: 23010
I20260925 02:23:59.868129 11764 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.125.20:38973
Sep 25, 2026 2:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:24:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 02:24:02 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:02.159366 11764 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.016s sys 0.171s
I20260925 02:24:02.159487 11764 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:24:02.159583 11764 mini_ranger_kms.cc:401] 127.11.125.20:38973/kms/v1/keys
Sep 25 02:24:02 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
Sep 25 02:24:02 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
Sep 25 02:24:05 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:24:05.245626 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38141
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:38141
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38973/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:05.345176 23061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:05.345412 23061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:05.345450 23061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:05.346611 23061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:05.346678 23061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:05.346704 23061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:05.346735 23061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:05.346768 23061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:05.348397 23061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.125.20:38973/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:38141
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:05.348668 23061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:05.348946 23061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:05.355216 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:05.355234 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:05.355312 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:05.355360 23061 server_base.cc:1034] running on GCE node
I20260925 02:24:05.355674 23061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:05.356032 23061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:05.357216 23061 hybrid_clock.cc:648] HybridClock initialized: now 1790303045357203 us; error 32 us; skew 500 ppm
Sep 25 02:24:05 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:05.366719 23061 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:24:05.367311 23061 webserver.cc:460] Webserver started at http://127.11.125.62:38007/ using document root <none> and password file <none>
I20260925 02:24:05.367590 23061 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:05.367648 23061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:05.367784 23061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:05 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303045, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:05.429993 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "7494dcafc3f44d51a9e7fc4d1140a738"
format_stamp: "Formatted at 2026-09-25 02:24:05 on dist-test-slave-hj7s"
server_key: "fd9c5d64a7a0511db827178b475fab48"
server_key_iv: "51539353a34272bdafa36bec108a73a8"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:05.430389 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "7494dcafc3f44d51a9e7fc4d1140a738"
format_stamp: "Formatted at 2026-09-25 02:24:05 on dist-test-slave-hj7s"
server_key: "fd9c5d64a7a0511db827178b475fab48"
server_key_iv: "51539353a34272bdafa36bec108a73a8"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:05.432040 23061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:24:05.447374 23077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:05.447731 23061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:24:05.447896 23061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "7494dcafc3f44d51a9e7fc4d1140a738"
format_stamp: "Formatted at 2026-09-25 02:24:05 on dist-test-slave-hj7s"
server_key: "fd9c5d64a7a0511db827178b475fab48"
server_key_iv: "51539353a34272bdafa36bec108a73a8"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:05.448004 23061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:05.461951 23061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:05.464092 23061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:05.464250 23061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:05.469247 23061 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:38141
I20260925 02:24:05.469273 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:38141 every 8 connection(s)
I20260925 02:24:05.469640 23061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:24:05.470357 23130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:05.472500 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 23061
I20260925 02:24:05.472610 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:24:05.472711 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: Bootstrap starting.
I20260925 02:24:05.472884 11764 external_mini_cluster.cc:1428] Setting key d7b6774e8d8a7b37920d3da16d758162
I20260925 02:24:05.473537 23130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:05.473973 23130 log.cc:824] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:05.474920 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: No bootstrap required, opened a new log
I20260925 02:24:05.477308 23130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } }
I20260925 02:24:05.477469 23130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:05.477531 23130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7494dcafc3f44d51a9e7fc4d1140a738, State: Initialized, Role: FOLLOWER
I20260925 02:24:05.477674 23130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } }
I20260925 02:24:05.477772 23130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:24:05.477835 23130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:24:05.477890 23130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:05.478602 23130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } }
I20260925 02:24:05.478758 23130 leader_election.cc:302] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7494dcafc3f44d51a9e7fc4d1140a738; no voters:
I20260925 02:24:05.478991 23130 leader_election.cc:288] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:24:05.479306 23130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:05.479337 23135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:24:05.480954 23135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [term 1 LEADER]: Becoming Leader. State: Replica: 7494dcafc3f44d51a9e7fc4d1140a738, State: Running, Role: LEADER
I20260925 02:24:05.481050 23135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } }
Sep 25 02:24:05 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:05.481392 23147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7494dcafc3f44d51a9e7fc4d1140a738. Latest consensus state: current_term: 1 leader_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } } }
I20260925 02:24:05.481676 23136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7494dcafc3f44d51a9e7fc4d1140a738" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38141 } } }
I20260925 02:24:05.481683 23130 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:24:05.481992 23136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:05.482335 23149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:24:05.482632 23147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:05.482852 23149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:24:05.484544 23149 catalog_manager.cc:1347] Generated new cluster ID: aa5a887dffe74c6aa181a9ae954d9e3a
I20260925 02:24:05.484598 23149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:24:05.489225 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:05.473543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.11.125.62:38141)
0925 02:24:05.473763 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:05.473775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:05.473813 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:24:05.474452 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:05.474461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:05.474545 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:05.474728 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:05.475459 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:05.476941 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:05.477965 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:05.478589 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:05.478770 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:05.481729 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:05.481733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:05.481733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:05.481745 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:05.484251 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:05.486506 (+ 2255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:05.486560 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:05.486560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:05.486993 (+ 433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:05.487219 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:05.487220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:05.487221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:05.487880 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:05.488161 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:05.488225 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:24:05.488852 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
I20260925 02:24:05.496258 23149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:24:05.497159 23149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:24:05.501992 23149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: Generated new TSK 0
I20260925 02:24:05.502296 23149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:24:06.289914 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:06.404707 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:06.404958 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:06.405010 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:06.406243 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:06.406287 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:06.406313 23175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:24:06.407809 23175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.125.20:38973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:06.408063 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:06.408334 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:06.414831 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.414870 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.415009 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:06.416162 23175 server_base.cc:1034] running on GCE node
I20260925 02:24:06.416455 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:06.416795 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:06.418076 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790303046418057 us; error 62 us; skew 500 ppm
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:06.421125 23175 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:24:06.421692 23175 webserver.cc:460] Webserver started at http://127.11.125.1:34599/ using document root <none> and password file <none>
I20260925 02:24:06.421943 23175 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:06.421990 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:06.422124 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:06.442544 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "61b86fdc0784c032ef9c9fe8c830372a"
server_key_iv: "a57d6e80dffc7e02816587ae7e2c6357"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.443169 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "61b86fdc0784c032ef9c9fe8c830372a"
server_key_iv: "a57d6e80dffc7e02816587ae7e2c6357"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.449410 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260925 02:24:06.471678 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.471952 23175 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260925 02:24:06.472041 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "61b86fdc0784c032ef9c9fe8c830372a"
server_key_iv: "a57d6e80dffc7e02816587ae7e2c6357"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.472131 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:06.505569 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:06.508311 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:06.508524 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:06.508827 23175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:06.509285 23175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:06.509377 23175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.509442 23175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:06.509488 23175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.516088 23175 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:42201
I20260925 02:24:06.516603 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:24:06.517729 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 23175
I20260925 02:24:06.517840 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:24:06.518059 11764 external_mini_cluster.cc:1428] Setting key 4b9245f62daeea18c5b6b5c2e21a1d00
I20260925 02:24:06.534017 23306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:42201 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:06.544459 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:06.517864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38141 (local address 127.11.125.1:42315)
0925 02:24:06.520745 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:06.520795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:06.521457 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:06.529340 (+ 7883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:06.529347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:06.529690 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:06.530061 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:06.530070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.531312 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.531314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:06.532979 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:06.532983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.533650 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.533652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:06.533828 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:06.534533 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:06.534545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:06.539368 (+ 4823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:06.542324 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:06.542334 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:06.542338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:06.542744 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:06.542997 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:06.542998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:06.542999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:06.543173 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:06.543541 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:06.543549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:06.543789 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 02:24:06.543904 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:24:06.544104 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2829,"thread_start_us":93,"threads_started":1}
I20260925 02:24:06.545499 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:06.518183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42315 (local address 127.11.125.62:38141)
0925 02:24:06.528993 (+ 10810us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:06.528995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:06.529011 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:24:06.529047 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:06.529048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:06.529066 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:06.530189 (+ 1123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:06.530240 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.531165 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.533112 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.533515 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.534817 (+ 1302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:06.539535 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:06.539537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:06.539538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:06.539549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:06.542195 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:06.542848 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:06.542849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:06.542850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:06.542921 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:06.543262 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:06.543262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:06.543263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:06.543446 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:06.545166 (+ 1720us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:06.545222 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:24:06.545334 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10776,"thread_start_us":60,"threads_started":1}
I20260925 02:24:06.546226 23307 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38141
I20260925 02:24:06.546344 23307 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:06.546578 23307 heartbeater.cc:511] Master 127.11.125.62:38141 requested a full tablet report, sending...
I20260925 02:24:06.547214 23094 ts_manager.cc:194] Registered new tserver with Master: 17d2b9aebfa444f9bcf0ad24a4b6900c (127.11.125.1:42201)
I20260925 02:24:06.548285 23094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:42315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:24:06.558094 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:24:06.683600 23315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:06.684031 23315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:06.684068 23315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:06.685902 23315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:06.685961 23315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:06.685998 23315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:24:06.688118 23315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.125.20:38973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:06.688354 23315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:06.688613 23315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:06.695042 23321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.695109 23322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.695317 23324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:06.695788 23315 server_base.cc:1034] running on GCE node
I20260925 02:24:06.696012 23315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:06.696300 23315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:06.697482 23315 hybrid_clock.cc:648] HybridClock initialized: now 1790303046697451 us; error 45 us; skew 500 ppm
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:06.700892 23315 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:24:06.701529 23315 webserver.cc:460] Webserver started at http://127.11.125.2:38123/ using document root <none> and password file <none>
I20260925 02:24:06.701798 23315 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:06.701860 23315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:06.701987 23315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:06.715646 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "cf7996aba1174382a8ec51dbe8ccb744"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "3db5596a088bf5c6b51816a1ee27dc63"
server_key_iv: "71d0cbe4bfcf60a7935bfd9ca459a1bd"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.716154 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "cf7996aba1174382a8ec51dbe8ccb744"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "3db5596a088bf5c6b51816a1ee27dc63"
server_key_iv: "71d0cbe4bfcf60a7935bfd9ca459a1bd"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.718252 23315 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 02:24:06.753036 23331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.753262 23315 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260925 02:24:06.753387 23315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "cf7996aba1174382a8ec51dbe8ccb744"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "3db5596a088bf5c6b51816a1ee27dc63"
server_key_iv: "71d0cbe4bfcf60a7935bfd9ca459a1bd"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.753515 23315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:06.787756 23315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:06.789944 23315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:06.790061 23315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:06.790373 23315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:06.790728 23315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:06.790783 23315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.790817 23315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:06.790846 23315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:06.798635 23315 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:33167
I20260925 02:24:06.798713 23445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:33167 every 8 connection(s)
I20260925 02:24:06.799026 23315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:06.806226 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 23315
I20260925 02:24:06.806382 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:24:06.806594 11764 external_mini_cluster.cc:1428] Setting key 179f734022a1dfec9f323c8bc40df649
I20260925 02:24:06.808736 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:06.799822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:52021 (local address 127.11.125.62:38141)
0925 02:24:06.799874 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:06.799876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:06.800102 (+ 226us) server_negotiation.cc:407] Connection header received
0925 02:24:06.800647 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:06.800648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:06.800666 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:06.800718 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:06.801520 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.802026 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.803056 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.803292 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.803379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:06.805564 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:06.805566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:06.805566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:06.805593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:06.807226 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:06.807720 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:06.807722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:06.807722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:06.807782 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:06.808043 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:06.808044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:06.808044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:06.808216 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:06.808276 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:06.808560 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:24:06.808649 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:24:06.808982 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:06.799823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38141 (local address 127.11.125.2:52021)
0925 02:24:06.799982 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:06.800023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:06.800525 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:06.800789 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:06.800793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:06.801080 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:06.801370 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:06.801377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.802123 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.802124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:06.802929 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:06.802931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:06.803398 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:06.803399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:06.803536 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:06.804065 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:06.804073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:06.805474 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:06.807322 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:06.807327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:06.807329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:06.807628 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:06.807854 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:06.807855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:06.807856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:06.807971 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:06.808278 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:06.808285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:06.808424 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 02:24:06.808528 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:24:06.808681 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260925 02:24:06.809509 23446 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38141
I20260925 02:24:06.809664 23446 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:06.809911 23446 heartbeater.cc:511] Master 127.11.125.62:38141 requested a full tablet report, sending...
I20260925 02:24:06.810426 23094 ts_manager.cc:194] Registered new tserver with Master: cf7996aba1174382a8ec51dbe8ccb744 (127.11.125.2:33167)
I20260925 02:24:06.810913 23094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:52021
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:24:06.837790 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38973/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:06.947515 23451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:06.947759 23451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:06.947798 23451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:06.949142 23451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:06.949236 23451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:06.949296 23451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:24:06.951170 23451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.125.20:38973/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:06.951476 23451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:06.951722 23451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:06.956871 23456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.956872 23459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:06.957224 23457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:06.959067 23451 server_base.cc:1034] running on GCE node
I20260925 02:24:06.959369 23451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:06.959651 23451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:06.960806 23451 hybrid_clock.cc:648] HybridClock initialized: now 1790303046960784 us; error 40 us; skew 500 ppm
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:06.963840 23451 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:24:06.964395 23451 webserver.cc:460] Webserver started at http://127.11.125.3:40929/ using document root <none> and password file <none>
I20260925 02:24:06.964700 23451 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:06.964767 23451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:06.964903 23451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:06 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:06.977362 23451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "0f730d892ce8428c8170db6ef05ff86d"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "659c5709799b90627c7b74bfed72d62b"
server_key_iv: "14e15a845f0f40a66591c3a0f993376b"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.977936 23451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "0f730d892ce8428c8170db6ef05ff86d"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "659c5709799b90627c7b74bfed72d62b"
server_key_iv: "14e15a845f0f40a66591c3a0f993376b"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:06.985378 23451 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 02:24:07.014298 23466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:07.014569 23451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 02:24:07.014668 23451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "0f730d892ce8428c8170db6ef05ff86d"
format_stamp: "Formatted at 2026-09-25 02:24:06 on dist-test-slave-hj7s"
server_key: "659c5709799b90627c7b74bfed72d62b"
server_key_iv: "14e15a845f0f40a66591c3a0f993376b"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:07.014767 23451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:07.038833 23451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:07.041409 23451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:07.041582 23451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:07.041851 23451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:07.042309 23451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:07.042373 23451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:07.042413 23451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:07.042439 23451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:07.056982 23451 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:39547
I20260925 02:24:07.057174 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:39547 every 8 connection(s)
I20260925 02:24:07.057830 23451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:24:07.065018 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 23451
I20260925 02:24:07.065150 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:24:07.065371 11764 external_mini_cluster.cc:1428] Setting key 4fb67d2353b1ba4856515e95c758fc01
Sep 25 02:24:07 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
Sep 25 02:24:07 dist-test-slave-hj7s krb5kdc[22614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303046, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:07.070061 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.059173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:53379 (local address 127.11.125.62:38141)
0925 02:24:07.059223 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.059226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.059236 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:24:07.060085 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.060087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.060100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.060168 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:07.060827 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.061468 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.062869 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.063136 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.063206 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.066702 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:07.066704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:07.066704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:07.066713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:07.068406 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:07.068914 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:07.068916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:07.068916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:07.069004 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:07.069346 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:07.069347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:07.069348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:07.069476 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:07.069558 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.069865 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.069960 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:24:07.070488 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.058874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38141 (local address 127.11.125.3:53379)
0925 02:24:07.059066 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.059113 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.059956 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.060248 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.060252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.060480 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:07.060709 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.060714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.061573 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.061574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.062768 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.062771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.063283 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.063285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.063439 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.064034 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:07.064043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:07.065801 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:07.068523 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:07.068530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:07.068533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:07.068810 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:07.069093 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:07.069094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:07.069094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:07.069252 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:07.069561 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:07.069569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:07.069715 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.069841 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.069999 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260925 02:24:07.071138 23581 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38141
I20260925 02:24:07.071244 23581 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:07.071465 23581 heartbeater.cc:511] Master 127.11.125.62:38141 requested a full tablet report, sending...
I20260925 02:24:07.071869 23094 ts_manager.cc:194] Registered new tserver with Master: 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547)
I20260925 02:24:07.072404 23094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:53379
I20260925 02:24:07.081640 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:24:07.092793 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.083565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.11.125.62:38141)
0925 02:24:07.083620 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.083622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.085105 (+ 1483us) server_negotiation.cc:407] Connection header received
0925 02:24:07.085203 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.085204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.085218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.085280 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:07.086118 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.086791 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.087761 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.087977 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.088062 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.089719 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:07.089721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:07.089721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:07.089730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:07.091391 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:07.091808 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:07.091809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:07.091810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:07.091874 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:07.092098 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:07.092099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:07.092099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:07.092225 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:07.092345 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.092547 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.092648 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:24:07.094818 23094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:24:07.097357 23094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:24:07.119467 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42201 (local address 127.0.0.1:51268)
0925 02:24:07.111587 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.111594 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.111627 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.111904 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.111908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.111910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:07.111976 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.111978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.113607 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.113608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.114830 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.114833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.119130 (+ 4297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.119131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.119228 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.119245 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.119319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.119373 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":71,"threads_started":1}
I20260925 02:24:07.119809 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.11.125.1:42201)
0925 02:24:07.111166 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.111172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.111678 (+ 506us) server_negotiation.cc:407] Connection header received
0925 02:24:07.111722 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.111724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.111739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.111814 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:07.112045 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.113542 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.114926 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.118193 (+ 3267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.118263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.118324 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.119673 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.119722 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":59,"threads_started":1}
I20260925 02:24:07.120527 23241 tablet_service.cc:1467] Processing CreateTablet for tablet a58bf98eeeca4efca631933734c455bc (DEFAULT_TABLE table=test-table [id=e55df4cf82b3415593ca6b559ff8724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:07.120950 23241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58bf98eeeca4efca631933734c455bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:07.121914 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:33167 (local address 127.0.0.1:40036)
0925 02:24:07.111234 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.111283 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.111367 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.113252 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.113253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.113255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:07.113303 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.113305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.116658 (+ 3353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.116659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.117730 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.117733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.121683 (+ 3950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.121684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.121750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.121754 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.121806 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.121847 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":72,"threads_started":1}
I20260925 02:24:07.122555 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.11.125.2:33167)
0925 02:24:07.113016 (+ 2272us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.113022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.113032 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:24:07.113074 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.113076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.113091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.113173 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:07.115530 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.116562 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.119942 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.121503 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.121575 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.121646 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.122426 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.122470 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2233,"thread_start_us":81,"threads_started":1}
I20260925 02:24:07.123204 23380 tablet_service.cc:1467] Processing CreateTablet for tablet a58bf98eeeca4efca631933734c455bc (DEFAULT_TABLE table=test-table [id=e55df4cf82b3415593ca6b559ff8724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:07.123551 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58bf98eeeca4efca631933734c455bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:07.124519 23600 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: Bootstrap starting.
I20260925 02:24:07.125411 23600 tablet_bootstrap.cc:654] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:07.125800 23600 log.cc:824] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:07.126585 23600 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: No bootstrap required, opened a new log
I20260925 02:24:07.126678 23600 ts_tablet_manager.cc:1397] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:24:07.126983 23602 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Bootstrap starting.
I20260925 02:24:07.128229 23602 tablet_bootstrap.cc:654] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:07.128661 23602 log.cc:824] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:07.133541 23600 raft_consensus.cc:348] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.133746 23600 raft_consensus.cc:374] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:07.133796 23600 raft_consensus.cc:729] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d2b9aebfa444f9bcf0ad24a4b6900c, State: Initialized, Role: FOLLOWER
I20260925 02:24:07.133924 23600 consensus_queue.cc:260] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.134238 23600 ts_tablet_manager.cc:1428] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c: Time spent starting tablet: real 0.008s user 0.000s sys 0.004s
I20260925 02:24:07.134371 23307 heartbeater.cc:503] Master 127.11.125.62:38141 was elected leader, sending a full tablet report...
I20260925 02:24:07.136291 23602 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: No bootstrap required, opened a new log
I20260925 02:24:07.136409 23602 ts_tablet_manager.cc:1397] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 02:24:07.137043 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39547 (local address 127.0.0.1:32960)
0925 02:24:07.111464 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.111472 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.111502 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.118652 (+ 7150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.118653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.118655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:07.118693 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.118695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.127283 (+ 8588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.127284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.128465 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.128469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.129724 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.129725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.129785 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.129789 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.129833 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.129873 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":65,"threads_started":1}
I20260925 02:24:07.137580 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.110905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.11.125.3:39547)
0925 02:24:07.118432 (+ 7527us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.118437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.118446 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:24:07.118483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.118485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.118501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.118572 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:07.118767 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.122333 (+ 3566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.128735 (+ 6402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.129544 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.129614 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.129688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.137396 (+ 7708us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.137467 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7479,"thread_start_us":64,"threads_started":1}
I20260925 02:24:07.138568 23498 tablet_service.cc:1467] Processing CreateTablet for tablet a58bf98eeeca4efca631933734c455bc (DEFAULT_TABLE table=test-table [id=e55df4cf82b3415593ca6b559ff8724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:07.139005 23498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58bf98eeeca4efca631933734c455bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:07.143579 23602 raft_consensus.cc:348] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.143791 23602 raft_consensus.cc:374] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:07.144086 23602 raft_consensus.cc:729] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7996aba1174382a8ec51dbe8ccb744, State: Initialized, Role: FOLLOWER
I20260925 02:24:07.144276 23602 consensus_queue.cc:260] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.144554 23606 raft_consensus.cc:482] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:24:07.144726 23606 raft_consensus.cc:504] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.145015 23602 ts_tablet_manager.cc:1428] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Time spent starting tablet: real 0.009s user 0.001s sys 0.003s
I20260925 02:24:07.145165 23446 heartbeater.cc:503] Master 127.11.125.62:38141 was elected leader, sending a full tablet report...
I20260925 02:24:07.145377 23606 leader_election.cc:288] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf7996aba1174382a8ec51dbe8ccb744 (127.11.125.2:33167), 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547)
I20260925 02:24:07.148943 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.145999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:46237 (local address 127.11.125.2:33167)
0925 02:24:07.146046 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.146048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.146057 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:24:07.146081 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.146082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.146094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.146156 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:07.146406 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.147047 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.148090 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.148512 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.148618 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.148629 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.148803 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.148843 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:24:07.148962 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.145526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:33167 (local address 127.11.125.1:46237)
0925 02:24:07.145855 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.145867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.145923 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.146210 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.146211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.146213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:07.146310 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.146312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.147160 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.147161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.147982 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.147984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.148609 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.148610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.148701 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.148705 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.148792 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.148840 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":126,"threads_started":1}
I20260925 02:24:07.149595 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58bf98eeeca4efca631933734c455bc" candidate_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7996aba1174382a8ec51dbe8ccb744" is_pre_election: true
I20260925 02:24:07.149760 23400 raft_consensus.cc:2454] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d2b9aebfa444f9bcf0ad24a4b6900c in term 0.
I20260925 02:24:07.150413 23194 leader_election.cc:302] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d2b9aebfa444f9bcf0ad24a4b6900c, cf7996aba1174382a8ec51dbe8ccb744; no voters:
I20260925 02:24:07.151541 23604 raft_consensus.cc:2781] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:24:07.151640 23604 raft_consensus.cc:482] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:24:07.151678 23604 raft_consensus.cc:3037] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:07.152472 23604 raft_consensus.cc:504] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.152779 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58bf98eeeca4efca631933734c455bc" candidate_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7996aba1174382a8ec51dbe8ccb744"
I20260925 02:24:07.152860 23400 raft_consensus.cc:3037] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:07.153918 23400 raft_consensus.cc:2454] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d2b9aebfa444f9bcf0ad24a4b6900c in term 1.
I20260925 02:24:07.154076 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.145733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:53409 (local address 127.11.125.3:39547)
0925 02:24:07.145780 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.145782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.149054 (+ 3272us) server_negotiation.cc:407] Connection header received
0925 02:24:07.149189 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.149191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.149202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.149261 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:07.149559 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.150209 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.152767 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.153396 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.153455 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.153465 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.153940 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.153992 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:24:07.154225 23194 leader_election.cc:302] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d2b9aebfa444f9bcf0ad24a4b6900c, cf7996aba1174382a8ec51dbe8ccb744; no voters:
I20260925 02:24:07.154309 23606 raft_consensus.cc:2781] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:24:07.154493 23606 raft_consensus.cc:686] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [term 1 LEADER]: Becoming Leader. State: Replica: 17d2b9aebfa444f9bcf0ad24a4b6900c, State: Running, Role: LEADER
I20260925 02:24:07.154616 23606 consensus_queue.cc:237] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.154999 23605 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Bootstrap starting.
I20260925 02:24:07.155514 23093 catalog_manager.cc:5495] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d2b9aebfa444f9bcf0ad24a4b6900c (127.11.125.1). New cstate: current_term: 1 leader_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } health_report { overall_health: HEALTHY } } }
I20260925 02:24:07.156841 23604 leader_election.cc:288] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 election: Requested vote from peers cf7996aba1174382a8ec51dbe8ccb744 (127.11.125.2:33167), 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547)
I20260925 02:24:07.156862 23605 tablet_bootstrap.cc:654] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:07.157006 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.145674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39547 (local address 127.11.125.1:53409)
0925 02:24:07.148976 (+ 3302us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.148990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:07.149082 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:07.149310 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:07.149312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:07.149314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:07.149390 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.149392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.150391 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.150392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.151364 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:07.151367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.153744 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.153746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:07.153836 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.153841 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:24:07.156658 (+ 2817us) client_negotiation.cc:240] Negotiation successful
0925 02:24:07.156707 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3281,"thread_start_us":115,"threads_started":1}
I20260925 02:24:07.157347 23605 log.cc:824] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:07.157387 23534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58bf98eeeca4efca631933734c455bc" candidate_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f730d892ce8428c8170db6ef05ff86d"
I20260925 02:24:07.157352 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58bf98eeeca4efca631933734c455bc" candidate_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f730d892ce8428c8170db6ef05ff86d" is_pre_election: true
W20260925 02:24:07.159204 23193 leader_election.cc:341] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547): Illegal state: must be running to vote when last-logged opid is not known
W20260925 02:24:07.159348 23193 leader_election.cc:341] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547): Illegal state: must be running to vote when last-logged opid is not known
I20260925 02:24:07.159891 23605 tablet_bootstrap.cc:492] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: No bootstrap required, opened a new log
I20260925 02:24:07.160029 23605 ts_tablet_manager.cc:1397] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:24:07.161558 23605 raft_consensus.cc:348] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.161769 23605 raft_consensus.cc:374] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:07.161867 23605 raft_consensus.cc:729] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f730d892ce8428c8170db6ef05ff86d, State: Initialized, Role: FOLLOWER
I20260925 02:24:07.162068 23605 consensus_queue.cc:260] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 } } peers { permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 } } peers { permanent_uuid: "17d2b9aebfa444f9bcf0ad24a4b6900c" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42201 } }
I20260925 02:24:07.162431 23605 ts_tablet_manager.cc:1428] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:24:07.162606 23581 heartbeater.cc:503] Master 127.11.125.62:38141 was elected leader, sending a full tablet report...
I20260925 02:24:07.163525 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.159969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.11.125.1:42201)
0925 02:24:07.160009 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:07.160011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:07.160072 (+ 61us) server_negotiation.cc:407] Connection header received
0925 02:24:07.160169 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:07.160170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:07.160188 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:07.160268 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:24:07.160460 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.161281 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.162496 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:07.162808 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:07.162881 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:07.163098 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:24:07.163225 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:24:07.163305 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:07.163393 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:24:07.163414 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:24:07.165233 23535 raft_consensus.cc:3037] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:07.165318 23400 raft_consensus.cc:1261] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 1 FOLLOWER]: Refusing update from remote peer 17d2b9aebfa444f9bcf0ad24a4b6900c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:07.166034 23535 raft_consensus.cc:1261] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 1 FOLLOWER]: Refusing update from remote peer 17d2b9aebfa444f9bcf0ad24a4b6900c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:07.166412 23606 consensus_queue.cc:1035] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f730d892ce8428c8170db6ef05ff86d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:07.166564 23604 consensus_queue.cc:1035] T a58bf98eeeca4efca631933734c455bc P 17d2b9aebfa444f9bcf0ad24a4b6900c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7996aba1174382a8ec51dbe8ccb744" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:07.169426 23615 mvcc.cc:205] Tried to move back new op lower bound from 7333081281186484224 to 7333081281146167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333081281186484224}]
I20260925 02:24:07.169638 23617 mvcc.cc:205] Tried to move back new op lower bound from 7333081281186484224 to 7333081281146167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333081281186484224}]
I20260925 02:24:07.171432 23618 mvcc.cc:205] Tried to move back new op lower bound from 7333081281186484224 to 7333081281146167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333081281186484224}]
I20260925 02:24:07.174134 23094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:24:07.174237 23094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:24:07.175040 23094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7494dcafc3f44d51a9e7fc4d1140a738: Sending DeleteTablet for 3 replicas of tablet a58bf98eeeca4efca631933734c455bc
I20260925 02:24:07.175588 23498 tablet_service.cc:1514] Processing DeleteTablet for tablet a58bf98eeeca4efca631933734c455bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:24:07 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:32960
I20260925 02:24:07.175562 23241 tablet_service.cc:1514] Processing DeleteTablet for tablet a58bf98eeeca4efca631933734c455bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:24:07 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:51268
I20260925 02:24:07.175809 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 23175
I20260925 02:24:07.175822 23380 tablet_service.cc:1514] Processing DeleteTablet for tablet a58bf98eeeca4efca631933734c455bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:24:07 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:40036
I20260925 02:24:07.176640 23628 tablet_replica.cc:330] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: stopping tablet replica
I20260925 02:24:07.176640 23626 tablet_replica.cc:330] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: stopping tablet replica
I20260925 02:24:07.176746 23626 raft_consensus.cc:2229] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:24:07.176745 23628 raft_consensus.cc:2229] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:24:07.176854 23626 raft_consensus.cc:2258] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:24:07.176860 23628 raft_consensus.cc:2258] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:24:07.177372 23626 ts_tablet_manager.cc:1905] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:24:07.177403 23628 ts_tablet_manager.cc:1905] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:24:07.179374 23628 ts_tablet_manager.cc:1918] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:24:07.179451 23628 log.cc:1196] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/a58bf98eeeca4efca631933734c455bc
I20260925 02:24:07.179715 23626 ts_tablet_manager.cc:1918] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:24:07.179803 23626 log.cc:1196] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/a58bf98eeeca4efca631933734c455bc
I20260925 02:24:07.180157 23626 ts_tablet_manager.cc:1939] T a58bf98eeeca4efca631933734c455bc P 0f730d892ce8428c8170db6ef05ff86d: Deleting consensus metadata
I20260925 02:24:07.180260 23628 ts_tablet_manager.cc:1939] T a58bf98eeeca4efca631933734c455bc P cf7996aba1174382a8ec51dbe8ccb744: Deleting consensus metadata
I20260925 02:24:07.180611 23078 catalog_manager.cc:4841] TS 0f730d892ce8428c8170db6ef05ff86d (127.11.125.3:39547): tablet a58bf98eeeca4efca631933734c455bc (table test-table [id=e55df4cf82b3415593ca6b559ff8724c]) successfully deleted
I20260925 02:24:07.180636 23079 catalog_manager.cc:4841] TS cf7996aba1174382a8ec51dbe8ccb744 (127.11.125.2:33167): tablet a58bf98eeeca4efca631933734c455bc (table test-table [id=e55df4cf82b3415593ca6b559ff8724c]) successfully deleted
W20260925 02:24:07.185070 23081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:42201 (error 108)
I20260925 02:24:07.185353 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 23315
I20260925 02:24:07.189210 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:07.189048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42201 (local address 127.0.0.1:51288)
0925 02:24:07.189121 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:07.189169 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:42201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:24:07.189348 23081 catalog_manager.cc:4584] TS 17d2b9aebfa444f9bcf0ad24a4b6900c (127.11.125.1:42201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a58bf98eeeca4efca631933734c455bc: Network error: Client connection negotiation failed: client connection to 127.11.125.1:42201: connect: Connection refused (error 111)
I20260925 02:24:07.191032 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 23451
I20260925 02:24:07.197877 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 23061
I20260925 02:24:07.203030 11764 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:24:07.547410 11764 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:24:07.547566 11764 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:24:07.892439 11764 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:24:07.892 UTC [22644] LOG: received smart shutdown request
2026-09-25 02:24:07.896 UTC [22644] LOG: background worker "logical replication launcher" (PID 22653) exited with exit code 1
2026-09-25 02:24:07.896 UTC [22648] LOG: shutting down
2026-09-25 02:24:08.073 UTC [22644] LOG: database system is shut down
2026-09-25T02:24:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45973 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:24:08 dist-test-slave-hj7s krb5kdc[23638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:24:08 dist-test-slave-hj7s krb5kdc[23638](info): set up 2 sockets
Sep 25 02:24:08 dist-test-slave-hj7s krb5kdc[23638](info): commencing operation
krb5kdc: starting...
W20260925 02:24:09.235239 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:24:09 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:24:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangeradmin_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangeradmin_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangerlookup_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangerlookup_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
I20260925 02:24:09.404242 11764 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task70c4Wq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/postgres -l logfile start
2026-09-25 02:24:12.040 UTC [23668] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:24:12.040 UTC [23668] LOG: listening on IPv4 address "127.11.125.20", port 53971
2026-09-25 02:24:12.044 UTC [23668] LOG: listening on Unix socket "/tmp/.s.PGSQL.53971"
2026-09-25 02:24:12.074 UTC [23671] LOG: database system was shut down at 2026-09-25 02:24:10 UTC
2026-09-25 02:24:12.077 UTC [23668] LOG: database system is ready to accept connections
I20260925 02:24:13.047703 11764 mini_postgres.cc:96] Postgres bound to 53971
2026-09-25 02:24:13.054 UTC [23679] FATAL: database "slave" does not exist
127.11.125.20:53971 - accepting connections
I20260925 02:24:13.054636 11764 mini_ranger.cc:161] Starting Ranger...
I20260925 02:24:13.069790 11764 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:24:13.598160 11764 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:24:13.598248 11764 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin
2026-09-25 02:24:13,876 [I] DB FLAVOR :POSTGRES
2026-09-25 02:24:13,876 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:24:13,876 [I] Checking connection..
2026-09-25 02:24:13,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:24:14,111 [I] Checking connection passed.
2026-09-25 02:24:14,111 [I] --------- Verifying version history table ---------
2026-09-25 02:24:14,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:24:14,357 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:24:14,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:24:14,591 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:24:14,591 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:24:14,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:24:14.823 UTC [23743] WARNING: there is no transaction in progress
2026-09-25 02:24:14,829 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:24:14,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:24:15,073 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:24:15,073 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:24:15,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:24:15,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:24:15,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hj7s.c.gcp-upstream.internal','N') ;"
2026-09-25 02:24:15,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:24:16,109 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:24:16,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:24:16,375 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:24:16,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:24:16,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:24:16,865 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:24:16,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:24:17.836 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.853 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.870 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.886 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.905 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.924 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.982 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:17.990 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.006 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.016 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.027 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.041 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.051 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.059 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.066 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18.072 UTC [23851] WARNING: there is no transaction in progress
2026-09-25 02:24:18,766 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:24:18,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:24:18,992 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:24:18,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:24:19,256 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:24:19,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:24:19,523 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:24:19,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:24:19,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:24:20,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hj7s.c.gcp-upstream.internal';"
2026-09-25 02:24:20,279 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:24:20,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:24:20,552 [I] DB_PATCHES have already been applied
I20260925 02:24:20.558640 11764 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task70c4Wq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:24:20.558727 11764 mini_ranger.cc:193] Using host: 127.11.125.20
I20260925 02:24:20.561976 11764 mini_ranger.cc:227] Ranger admin URL: http://127.11.125.20:56107
Sep 25, 2026 2:24:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:24:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:24:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:24:21.690917 11764 mini_ranger.cc:160] Time spent starting Ranger: real 8.636s user 0.003s sys 0.006s
Sep 25, 2026 2:24:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:24:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:24:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:24:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:24:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:24:44.500981 11764 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.125.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.125.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangerkms_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.125.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/rangerkms_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.125.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.125.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.125.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/HTTP_127.11.125.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:24:44 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:24:44.585974 11764 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:24:44.595347 11764 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:24:45.484107 11764 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:24:45.484215 11764 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms
I20260925 02:24:45.484263 11764 mini_ranger_kms.cc:227] Using postgres at 127.11.125.20:53971
2026-09-25 02:24:45,659 [I] DB FLAVOR :POSTGRES
2026-09-25 02:24:45,659 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:24:45,660 [I] Checking connection
2026-09-25 02:24:45,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:24:45,921 [I] connection success
2026-09-25 02:24:45,922 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:24:45,922 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:24:45,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:24:46,163 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:24:46,163 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:24:46,163 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:24:46,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.125.20:53971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:24:46,430 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:24:46.939155 11764 mini_ranger_kms.cc:310] Created kms service
I20260925 02:24:47.142784 11764 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:24:47.189841 11764 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:24:47 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:47.659698 11764 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:24:47.659919 11764 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms:/tmp/dist-test-task70c4Wq/build/release/bin/postgresql.jar:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task70c4Wq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:24:47.659943 11764 mini_ranger_kms.cc:240] Using host: 127.11.125.20
I20260925 02:24:47.663468 11764 mini_ranger_kms.cc:276] Ranger KMS PID: 24032
I20260925 02:24:47.663558 11764 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.125.20:38307
Sep 25, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:24:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:24:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 02:24:49 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:49.953899 11764 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.368s user 0.002s sys 0.155s
I20260925 02:24:49.954020 11764 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:24:49.954069 11764 mini_ranger_kms.cc:401] 127.11.125.20:38307/kms/v1/keys
Sep 25 02:24:50 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
Sep 25 02:24:50 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.125.20@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
Sep 25 02:24:53 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:24:53.120610 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40427
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:40427
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:53.210969 24083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:53.211189 24083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:53.211211 24083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:53.212364 24083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:53.212404 24083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:53.212419 24083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:53.212430 24083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:53.212442 24083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:53.213972 24083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.125.20:38307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:40427
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:53.214166 24083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:53.214339 24083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:53.219772 24088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:53.219772 24089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:53.219919 24083 server_base.cc:1034] running on GCE node
W20260925 02:24:53.219836 24091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:53.220201 24083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:53.220522 24083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:53.221665 24083 hybrid_clock.cc:648] HybridClock initialized: now 1790303093221646 us; error 35 us; skew 500 ppm
Sep 25 02:24:53 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303093, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:53.224131 24083 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:24:53.224593 24083 webserver.cc:460] Webserver started at http://127.11.125.62:43589/ using document root <none> and password file <none>
I20260925 02:24:53.224815 24083 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:53.224858 24083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:53.225033 24083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:53 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303093, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:53.274219 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "6119ec7147db4279854f261d7d8f6161"
format_stamp: "Formatted at 2026-09-25 02:24:53 on dist-test-slave-hj7s"
server_key: "8237fa88955ebf0bab5f44a5df1c833b"
server_key_iv: "816600514fc041ff8ea3b9c240168211"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:53.274698 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "6119ec7147db4279854f261d7d8f6161"
format_stamp: "Formatted at 2026-09-25 02:24:53 on dist-test-slave-hj7s"
server_key: "8237fa88955ebf0bab5f44a5df1c833b"
server_key_iv: "816600514fc041ff8ea3b9c240168211"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:53.276616 24083 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:24:53.292520 24098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:53.292841 24083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:24:53.292990 24083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "6119ec7147db4279854f261d7d8f6161"
format_stamp: "Formatted at 2026-09-25 02:24:53 on dist-test-slave-hj7s"
server_key: "8237fa88955ebf0bab5f44a5df1c833b"
server_key_iv: "816600514fc041ff8ea3b9c240168211"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:53.293093 24083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:53.313232 24083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:53.315552 24083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:53.315704 24083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:53.321542 24083 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:40427
I20260925 02:24:53.321638 24151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:40427 every 8 connection(s)
I20260925 02:24:53.322036 24083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:24:53.322844 24152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:53.325959 24152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Bootstrap starting.
I20260925 02:24:53.326653 24152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:53.326959 24152 log.cc:824] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:53.327396 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24083
I20260925 02:24:53.327509 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:24:53.327672 11764 external_mini_cluster.cc:1428] Setting key a81dd0a2bf74952181756e8ff536a911
I20260925 02:24:53.327790 24152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: No bootstrap required, opened a new log
I20260925 02:24:53.330684 24152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:53.330856 24152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:53.330888 24152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6119ec7147db4279854f261d7d8f6161, State: Initialized, Role: FOLLOWER
I20260925 02:24:53.331063 24152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:53.331161 24152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:24:53.331221 24152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:24:53.331295 24152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:53.331977 24152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:53.332083 24152 leader_election.cc:302] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6119ec7147db4279854f261d7d8f6161; no voters:
I20260925 02:24:53.332278 24152 leader_election.cc:288] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:24:53.332515 24152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:53.334029 24152 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 02:24:53 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:53.334533 24158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:24:53.337848 24158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 LEADER]: Becoming Leader. State: Replica: 6119ec7147db4279854f261d7d8f6161, State: Running, Role: LEADER
I20260925 02:24:53.341248 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:53.328206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.11.125.62:40427)
0925 02:24:53.328544 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:53.328552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:53.328574 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:24:53.329130 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:53.329138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:53.329189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:53.329336 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:53.330020 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:53.330992 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:53.331893 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:53.332481 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:53.332604 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:53.336319 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:53.336359 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:53.336360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:53.336412 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:53.338857 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:53.339365 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:53.339368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:53.339368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:53.339461 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:53.339662 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:53.339663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:53.339663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:53.340214 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:53.340374 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:53.340818 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 02:24:53.341010 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":57,"threads_started":1}
I20260925 02:24:53.341668 24158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:53.342146 24158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6119ec7147db4279854f261d7d8f6161. Latest consensus state: current_term: 1 leader_uuid: "6119ec7147db4279854f261d7d8f6161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } } }
I20260925 02:24:53.342306 24158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:53.342610 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6119ec7147db4279854f261d7d8f6161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } } }
I20260925 02:24:53.342694 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:53.344111 24174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:24:53.344725 24174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:24:53.346678 24174 catalog_manager.cc:1347] Generated new cluster ID: fac0c6a9ac2642b8ad2bcbc74094e436
I20260925 02:24:53.346725 24174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:24:53.351890 24174 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:24:53.352895 24174 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:24:53.359977 24174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Generated new TSK 0
I20260925 02:24:53.360211 24174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:24:54.076068 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:54.162940 24198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:54.163169 24198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:54.163208 24198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:54.164471 24198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:54.164538 24198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:54.164582 24198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:24:54.166117 24198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:54.166386 24198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:54.166627 24198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:54.172791 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.172787 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.172967 24198 server_base.cc:1034] running on GCE node
W20260925 02:24:54.172788 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.173357 24198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:54.173645 24198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:54.174834 24198 hybrid_clock.cc:648] HybridClock initialized: now 1790303094174798 us; error 45 us; skew 500 ppm
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:54.177878 24198 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:24:54.178354 24198 webserver.cc:460] Webserver started at http://127.11.125.1:35765/ using document root <none> and password file <none>
I20260925 02:24:54.178596 24198 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:54.178664 24198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:54.178815 24198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:54.192731 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "8d07584da6241aa9ae0e1c378c696147"
server_key_iv: "76f7c285cbb3e93ba66720a97a3e6fa0"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.193305 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "8d07584da6241aa9ae0e1c378c696147"
server_key_iv: "76f7c285cbb3e93ba66720a97a3e6fa0"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.194775 24198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:54.207667 24213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.207896 24198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:54.207996 24198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "8d07584da6241aa9ae0e1c378c696147"
server_key_iv: "76f7c285cbb3e93ba66720a97a3e6fa0"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.208114 24198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:54.226404 24198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:54.228777 24198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:54.229007 24198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:54.229274 24198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:54.229733 24198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:54.229866 24198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.229974 24198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:54.230050 24198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.237135 24198 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:39929
I20260925 02:24:54.237664 24198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:24:54.238371 24329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:39929 every 8 connection(s)
I20260925 02:24:54.241850 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24198
I20260925 02:24:54.241967 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:24:54.242166 11764 external_mini_cluster.cc:1428] Setting key a72d72678c0e30838424361da6434b6d
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:24:54.256412 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.238778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:43287 (local address 127.11.125.62:40427)
0925 02:24:54.245038 (+ 6260us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:54.245041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:54.245598 (+ 557us) server_negotiation.cc:407] Connection header received
0925 02:24:54.246393 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:54.246395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:54.246411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:54.246461 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:54.247232 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.248064 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.249757 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.250073 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.250161 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.253029 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:54.253031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:54.253032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:54.253040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:54.254911 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.255330 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.255331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.255332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.255402 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.255657 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.255658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.255659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.255784 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:54.255869 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:54.256218 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:24:54.256312 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6172,"thread_start_us":96,"threads_started":1}
I20260925 02:24:54.256599 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.238764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.1:43287)
0925 02:24:54.245460 (+ 6696us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:54.245502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:54.246256 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:54.246678 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:54.246684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:54.246930 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:54.247133 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.247138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.248177 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.248178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.249646 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.249649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.250215 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.250216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.250395 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.251042 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:54.251053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:54.252823 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:54.255003 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.255010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.255012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.255241 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.255470 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.255471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.255471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.255582 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.255859 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:54.255865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:54.256056 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 02:24:54.256182 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:24:54.256330 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6652,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
I20260925 02:24:54.257189 24330 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:54.257283 24330 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:54.257489 24330 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:54.258039 24115 ts_manager.cc:194] Registered new tserver with Master: 3c2bfaf621564fe7a8361b4aa7c9aafa (127.11.125.1:39929)
I20260925 02:24:54.258905 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:43287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:24:54.272292 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:54.385406 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:54.385638 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:54.385680 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:54.386901 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:54.386976 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:54.387019 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:24:54.388561 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:54.388816 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:54.389091 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:54.393980 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.393980 24345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.394343 24347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.394874 24336 server_base.cc:1034] running on GCE node
I20260925 02:24:54.395094 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:54.395357 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:54.396543 24336 hybrid_clock.cc:648] HybridClock initialized: now 1790303094396531 us; error 30 us; skew 500 ppm
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:54.399415 24336 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:24:54.399906 24336 webserver.cc:460] Webserver started at http://127.11.125.2:39353/ using document root <none> and password file <none>
I20260925 02:24:54.400151 24336 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:54.400218 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:54.400373 24336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:54.413139 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "823580b626134903b28a98c6a9c38e01"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "cd6c88eb4dac878c628945da39e7f9f4"
server_key_iv: "6ddada7cd2661245ccc23cd006dc79af"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.413691 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "823580b626134903b28a98c6a9c38e01"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "cd6c88eb4dac878c628945da39e7f9f4"
server_key_iv: "6ddada7cd2661245ccc23cd006dc79af"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.416123 24336 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:54.432556 24354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.432799 24336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:54.432921 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "823580b626134903b28a98c6a9c38e01"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "cd6c88eb4dac878c628945da39e7f9f4"
server_key_iv: "6ddada7cd2661245ccc23cd006dc79af"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.433019 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:54.449515 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:54.452113 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:54.452332 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:54.452582 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:54.453040 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:54.453141 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.453235 24336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:54.453296 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.459703 24336 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:32841
I20260925 02:24:54.460201 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:24:54.462896 24467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:32841 every 8 connection(s)
I20260925 02:24:54.468743 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24336
I20260925 02:24:54.468865 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:24:54.469106 11764 external_mini_cluster.cc:1428] Setting key e746a2c16786ada648a36ff013cdd3de
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:54.475348 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.463747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:36733 (local address 127.11.125.62:40427)
0925 02:24:54.463797 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:54.463799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:54.464143 (+ 344us) server_negotiation.cc:407] Connection header received
0925 02:24:54.464994 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:54.464996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:54.465011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:54.465041 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:54.465868 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.466672 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.468243 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.468553 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.468600 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.471762 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:54.471764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:54.471764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:54.471772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:54.473664 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.474202 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.474204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.474204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.474265 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.474556 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.474557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.474558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.474698 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:54.474746 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:54.475130 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 02:24:54.475238 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:24:54.476121 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.463457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.2:36733)
0925 02:24:54.464014 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:54.464060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:54.464719 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:54.465240 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:54.465245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:54.465492 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:54.465761 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.465767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.466794 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.466795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.468122 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.468125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.468710 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.468711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.468988 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.469858 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:54.469869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:54.471654 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:54.473769 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.473776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.473778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.474097 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.474340 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.474340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.474341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.474473 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.474798 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:54.474805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:54.474967 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 02:24:54.475705 (+ 738us) client_negotiation.cc:240] Negotiation successful
0925 02:24:54.475877 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":86,"threads_started":1}
I20260925 02:24:54.476871 24468 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:54.476992 24468 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:54.477205 24468 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:54.477671 24115 ts_manager.cc:194] Registered new tserver with Master: 823580b626134903b28a98c6a9c38e01 (127.11.125.2:32841)
I20260925 02:24:54.478191 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:36733
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:24:54.507694 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:54.627853 24473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:54.628177 24473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:54.628234 24473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:54.629822 24473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:54.629923 24473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:54.629974 24473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:24:54.631603 24473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:54.631906 24473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:54.632220 24473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:54.637977 24479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.638111 24482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.637977 24480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.639160 24473 server_base.cc:1034] running on GCE node
I20260925 02:24:54.639384 24473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:54.639670 24473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:54.640851 24473 hybrid_clock.cc:648] HybridClock initialized: now 1790303094640832 us; error 27 us; skew 500 ppm
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:54.643461 24473 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:24:54.643980 24473 webserver.cc:460] Webserver started at http://127.11.125.3:43553/ using document root <none> and password file <none>
I20260925 02:24:54.644228 24473 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:54.644272 24473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:54.644413 24473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:54.658341 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "9c15470c987e43a6bc622ae93e69f2ab"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "5e0b9d2cb758567f2fd5b9727dd719c2"
server_key_iv: "df57fc8bfe0f0af92935e51a9be27215"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.658878 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "9c15470c987e43a6bc622ae93e69f2ab"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "5e0b9d2cb758567f2fd5b9727dd719c2"
server_key_iv: "df57fc8bfe0f0af92935e51a9be27215"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.661489 24473 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260925 02:24:54.679940 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.680166 24473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:24:54.680281 24473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "9c15470c987e43a6bc622ae93e69f2ab"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "5e0b9d2cb758567f2fd5b9727dd719c2"
server_key_iv: "df57fc8bfe0f0af92935e51a9be27215"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.680395 24473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:54.698284 24473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:54.701146 24473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:54.701323 24473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:54.701591 24473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:54.702030 24473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:54.702099 24473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.702159 24473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:54.702194 24473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.710765 24473 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:36005
I20260925 02:24:54.711262 24473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:24:54.712139 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:36005 every 8 connection(s)
I20260925 02:24:54.715796 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24473
I20260925 02:24:54.715922 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:24:54.716115 11764 external_mini_cluster.cc:1428] Setting key 7421b7069d727c5505ff935857fd33e8
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:54.724007 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.712802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:53239 (local address 127.11.125.62:40427)
0925 02:24:54.712855 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:54.712857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:54.713148 (+ 291us) server_negotiation.cc:407] Connection header received
0925 02:24:54.713753 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:54.713754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:54.713768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:54.713842 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:54.714644 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.715175 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.716379 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.716646 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.716699 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.720236 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:54.720238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:54.720239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:54.720248 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:54.721967 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.722545 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.722547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.722548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.722622 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:54.722934 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:54.722935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:54.722936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:54.723075 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:54.723174 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:54.723799 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 02:24:54.723905 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:24:54.725031 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:54.712767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.3:53239)
0925 02:24:54.712972 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:54.713018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:54.713619 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:54.713933 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:54.713938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:54.714218 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:54.714504 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.714513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.715282 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.715283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.716243 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:54.716247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:54.716827 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:54.716828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:54.716985 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:54.717653 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:54.717662 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:54.720102 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:54.722090 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.722099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.722101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.722441 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.722698 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:54.722699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:54.722700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:54.722854 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:54.723440 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:54.723446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:54.723623 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 02:24:54.723759 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:24:54.723930 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260925 02:24:54.725802 24604 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:54.725943 24604 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:54.726163 24604 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:54.726610 24115 ts_manager.cc:194] Registered new tserver with Master: 9c15470c987e43a6bc622ae93e69f2ab (127.11.125.3:36005)
I20260925 02:24:54.727072 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:53239
I20260925 02:24:54.728063 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:24:54.729116 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24198
I20260925 02:24:54.738155 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24336
I20260925 02:24:54.743173 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24473
I20260925 02:24:54.748024 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24083
I20260925 02:24:54.753674 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40427
--webserver_interface=127.11.125.62
--webserver_port=43589
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:40427
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:54.836588 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:54.836804 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:54.836827 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:54.838129 24607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:54.838172 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:54.838186 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:54.838198 24607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:54.838244 24607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:54.839845 24607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:40427
--ranger_config_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=43589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:54.840096 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:54.840323 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:54.845286 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:54.845331 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.845361 24607 server_base.cc:1034] running on GCE node
W20260925 02:24:54.845302 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:54.845755 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:54.846060 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:54.847232 24607 hybrid_clock.cc:648] HybridClock initialized: now 1790303094847224 us; error 42 us; skew 500 ppm
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:54.849777 24607 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:24:54.850180 24607 webserver.cc:460] Webserver started at http://127.11.125.62:43589/ using document root <none> and password file <none>
I20260925 02:24:54.850366 24607 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:54.850414 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:24:54 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303094, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:54.914539 24607 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:54.915531 24623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:54.915740 24607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:54.915838 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "6119ec7147db4279854f261d7d8f6161"
format_stamp: "Formatted at 2026-09-25 02:24:53 on dist-test-slave-hj7s"
server_key: "8237fa88955ebf0bab5f44a5df1c833b"
server_key_iv: "816600514fc041ff8ea3b9c240168211"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:54.916301 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:54.929385 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:54.931500 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:54.931661 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:54.936875 24607 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:40427
I20260925 02:24:54.936950 24676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:40427 every 8 connection(s)
I20260925 02:24:54.937417 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:24:54.938759 24677 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:24:54.939435 24677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Bootstrap starting.
I20260925 02:24:54.939672 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24607
I20260925 02:24:54.939975 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:39929
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:40427
--webserver_port=35765
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:24:54.948415 24677 log.cc:824] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:54.950304 24677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:24:54.950629 24677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Bootstrap complete.
I20260925 02:24:54.953150 24677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:54.953424 24677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6119ec7147db4279854f261d7d8f6161, State: Initialized, Role: FOLLOWER
I20260925 02:24:54.953627 24677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:54.953713 24677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:24:54.953768 24677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:24:54.953820 24677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:24:54.954859 24677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:54.954985 24677 leader_election.cc:302] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6119ec7147db4279854f261d7d8f6161; no voters:
I20260925 02:24:54.955190 24677 leader_election.cc:288] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:24:54.955229 24681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:24:54.955348 24681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [term 2 LEADER]: Becoming Leader. State: Replica: 6119ec7147db4279854f261d7d8f6161, State: Running, Role: LEADER
I20260925 02:24:54.955444 24681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } }
I20260925 02:24:54.955526 24677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:54.955710 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6119ec7147db4279854f261d7d8f6161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } } }
I20260925 02:24:54.955860 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:54.955708 24681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6119ec7147db4279854f261d7d8f6161. Latest consensus state: current_term: 2 leader_uuid: "6119ec7147db4279854f261d7d8f6161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6119ec7147db4279854f261d7d8f6161" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40427 } } }
I20260925 02:24:54.955958 24681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:54.957065 24686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:24:54.957121 24677 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:24:54.957559 24686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:24:54.957737 24686 catalog_manager.cc:1259] Loaded cluster ID: fac0c6a9ac2642b8ad2bcbc74094e436
I20260925 02:24:54.957791 24686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:24:54.958586 24686 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:24:54.958783 24686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Loaded TSK: 0
I20260925 02:24:54.959003 24686 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:24:55.062191 24679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:55.062497 24679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:55.062557 24679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:55.064353 24679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:55.064414 24679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:55.064455 24679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:24:55.066742 24679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:39929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=35765
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:55.067008 24679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:55.067265 24679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:55.073835 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:55.073982 24679 server_base.cc:1034] running on GCE node
W20260925 02:24:55.074188 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:55.074350 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:55.074647 24679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:55.074949 24679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:55.076148 24679 hybrid_clock.cc:648] HybridClock initialized: now 1790303095076102 us; error 51 us; skew 500 ppm
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:55.079504 24679 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:24:55.080083 24679 webserver.cc:460] Webserver started at http://127.11.125.1:35765/ using document root <none> and password file <none>
I20260925 02:24:55.080380 24679 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:55.080470 24679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:55.112422 24679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:55.113536 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.113793 24679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:24:55.113905 24679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "8d07584da6241aa9ae0e1c378c696147"
server_key_iv: "76f7c285cbb3e93ba66720a97a3e6fa0"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:55.114303 24679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:55.136865 24679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:55.139334 24679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:55.139523 24679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:55.139869 24679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:55.140295 24679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:55.140343 24679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.140376 24679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:55.140394 24679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.148102 24679 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:39929
I20260925 02:24:55.148272 24839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:39929 every 8 connection(s)
I20260925 02:24:55.148579 24679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:24:55.150033 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24679
I20260925 02:24:55.150347 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:32841
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:40427
--webserver_port=39353
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:55.163792 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.149595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.1:57951)
0925 02:24:55.149997 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:55.150046 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:55.151176 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:55.151955 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:55.151959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:55.152171 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:55.152371 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.152375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.154124 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.154126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.155491 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.155495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.156479 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.156481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.156638 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.157308 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:55.157319 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:55.158946 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:55.161572 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.161580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.161583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.161944 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.162288 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.162289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.162289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.162438 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.162992 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:55.162999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:55.163174 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 02:24:55.163277 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:24:55.163468 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":88,"threads_started":1}
I20260925 02:24:55.165042 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.149664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:57951 (local address 127.11.125.62:40427)
0925 02:24:55.149896 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:55.149908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:55.150239 (+ 331us) server_negotiation.cc:407] Connection header received
0925 02:24:55.151507 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:55.151515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:55.151585 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:55.151696 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:55.152500 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.153997 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.155631 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.156346 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.156983 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.159098 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:55.159103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:55.159103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:55.159117 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:55.161462 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.162069 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.162072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.162072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.162198 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.162528 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.162529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.162529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.162891 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:55.164492 (+ 1601us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:55.164577 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:24:55.164744 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260925 02:24:55.166188 24840 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:55.166299 24840 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.166492 24840 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.167410 24840 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:40427 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:24:55.167480 24840 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.167541 24840 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:55.167932 24840 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.168005 24840 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.168354 24840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:24:55.276798 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:55.277209 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:55.277309 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:55.279142 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:55.279266 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:55.279359 24844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:24:55.281773 24844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:32841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=39353
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:55.282248 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:55.282565 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:55.289601 24850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:55.290020 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:55.291167 24844 server_base.cc:1034] running on GCE node
W20260925 02:24:55.294960 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:55.295367 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:55.295687 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:55.296860 24844 hybrid_clock.cc:648] HybridClock initialized: now 1790303095296843 us; error 30 us; skew 500 ppm
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:55.300060 24844 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:24:55.300597 24844 webserver.cc:460] Webserver started at http://127.11.125.2:39353/ using document root <none> and password file <none>
I20260925 02:24:55.300859 24844 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:55.300959 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:55.325453 24844 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 02:24:55.333427 24860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.336280 24844 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260925 02:24:55.336428 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "823580b626134903b28a98c6a9c38e01"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "cd6c88eb4dac878c628945da39e7f9f4"
server_key_iv: "6ddada7cd2661245ccc23cd006dc79af"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:55.337025 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:55.395372 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:55.397994 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:55.398155 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:55.398419 24844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:55.398872 24844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:55.398967 24844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.399040 24844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:55.399097 24844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.405360 24844 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:32841
I20260925 02:24:55.405884 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:24:55.408427 24973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:32841 every 8 connection(s)
I20260925 02:24:55.411405 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24844
I20260925 02:24:55.411698 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.125.20:38307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:36005
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:40427
--webserver_port=43553
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:55.442090 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.409509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:38111 (local address 127.11.125.62:40427)
0925 02:24:55.409556 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:55.409558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:55.409570 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:24:55.410240 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:55.410241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:55.410256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:55.410287 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:55.411120 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.411948 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.413785 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.414099 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.414150 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.416873 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:55.416874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:55.416875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:55.416883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:55.418909 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.420312 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.420314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.420314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.420401 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.420791 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.420792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.420792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.420961 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:55.421014 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:55.441790 (+ 20776us) server_negotiation.cc:299] Negotiation successful
0925 02:24:55.441940 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:24:55.443090 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.409162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.2:38111)
0925 02:24:55.409377 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:55.409429 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:55.410101 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:55.410471 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:55.410476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:55.410698 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:55.410992 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.410999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.412291 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.412293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.413669 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.413672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.414277 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.414278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.414442 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.415086 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:55.415097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:55.416756 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:55.419044 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.419052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.419055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.419390 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.420489 (+ 1099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.420490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.420490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.420699 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.441233 (+ 20534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:55.441248 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:55.441552 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:24:55.442631 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0925 02:24:55.442843 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260925 02:24:55.447069 24974 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:55.447185 24974 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.447409 24974 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.448202 24974 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:40427 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:24:55.448278 24974 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.448355 24974 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:55.448743 24974 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.448818 24974 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.449190 24974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:24:55.529407 24977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:55.529764 24977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:55.529935 24977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:55.531600 24977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:55.531724 24977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:55.531811 24977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:24:55.534147 24977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.125.20:38307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:36005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=43553
--tserver_master_addrs=127.11.125.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:55.534651 24977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:55.534978 24977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:55.542011 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:55.542222 24983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:55.542047 24986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:55.543128 24977 server_base.cc:1034] running on GCE node
I20260925 02:24:55.543356 24977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:55.543661 24977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:55.544862 24977 hybrid_clock.cc:648] HybridClock initialized: now 1790303095544828 us; error 46 us; skew 500 ppm
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:55.557667 24977 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:24:55.558293 24977 webserver.cc:460] Webserver started at http://127.11.125.3:43553/ using document root <none> and password file <none>
I20260925 02:24:55.558624 24977 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:55.558741 24977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:55.577674 24977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:24:55.581415 24994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.581664 24977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260925 02:24:55.581766 24977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "9c15470c987e43a6bc622ae93e69f2ab"
format_stamp: "Formatted at 2026-09-25 02:24:54 on dist-test-slave-hj7s"
server_key: "5e0b9d2cb758567f2fd5b9727dd719c2"
server_key_iv: "df57fc8bfe0f0af92935e51a9be27215"
server_key_version: "kuduclusterkey@0"
I20260925 02:24:55.582293 24977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:55.600270 24977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:55.603582 24977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:55.603820 24977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:55.604195 24977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:24:55.604640 24977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:24:55.604733 24977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.604806 24977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:24:55.604876 24977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:55.611869 24977 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:36005
I20260925 02:24:55.612448 24977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:24:55.614878 25109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:36005 every 8 connection(s)
I20260925 02:24:55.619931 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 24977
Sep 25 02:24:55 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303095, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:55.627075 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.615636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:35529 (local address 127.11.125.62:40427)
0925 02:24:55.615685 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:55.615687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:55.615988 (+ 301us) server_negotiation.cc:407] Connection header received
0925 02:24:55.616615 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:55.616617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:55.616631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:55.616706 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:55.617526 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.618287 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.619886 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.620194 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.620246 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.623323 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:55.623325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:55.623325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:55.623333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:55.625095 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.625908 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.625909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.625909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.625973 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.626281 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.626282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.626282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.626418 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:55.626463 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:55.626872 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 02:24:55.626959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:24:55.627599 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.615622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40427 (local address 127.11.125.3:35529)
0925 02:24:55.615826 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:55.615876 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:55.616491 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:55.616803 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:55.616808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:55.617120 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:55.617410 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.617417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.618433 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.618435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.619764 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:55.619769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.620381 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.620382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:55.620530 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.621221 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:55.621233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:55.623196 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:55.625189 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.625198 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.625200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.625795 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.626051 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:55.626052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:55.626052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:55.626193 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:55.626513 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:55.626519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:55.626693 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 02:24:55.627184 (+ 491us) client_negotiation.cc:240] Negotiation successful
0925 02:24:55.627358 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260925 02:24:55.628211 25110 heartbeater.cc:347] Connected to a master server at 127.11.125.62:40427
I20260925 02:24:55.628316 25110 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.628515 25110 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.629141 25110 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:40427 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:24:55.629213 25110 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.629269 25110 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:55.629587 25110 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:55.629652 25110 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
W20260925 02:24:55.629927 25110 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:24:55.632164 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:55.620560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.11.125.62:40427)
0925 02:24:55.621278 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:55.621280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:55.621286 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:24:55.621306 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:55.621307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:55.621318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:55.621372 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:55.622228 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.623081 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.624740 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:55.625068 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:55.625124 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:55.626642 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:55.626644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:55.626644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:55.626651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:55.630670 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.631052 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.631053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.631053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.631107 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:55.631295 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:55.631296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:55.631296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:55.631427 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:55.631594 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:55.631985 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:24:55.632063 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":50,"threads_started":1}
Sep 25 02:24:56 dist-test-slave-hj7s krb5kdc[23638](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790303096, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:56.168540 24840 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:56.168715 24840 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:56.169126 24640 ts_manager.cc:194] Registered new tserver with Master: 3c2bfaf621564fe7a8361b4aa7c9aafa (127.11.125.1:39929)
I20260925 02:24:56.170153 24640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:57951
I20260925 02:24:56.449375 24974 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:56.449544 24974 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:56.450001 24640 ts_manager.cc:194] Registered new tserver with Master: 823580b626134903b28a98c6a9c38e01 (127.11.125.2:32841)
I20260925 02:24:56.450531 24640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:38111
Sep 25 02:24:56 dist-test-slave-hj7s krb5kdc[23638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303096, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for HTTP/127.11.125.20@KRBTEST.COM
I20260925 02:24:56.630112 25110 heartbeater.cc:464] Registering TS with master...
I20260925 02:24:56.630303 25110 heartbeater.cc:511] Master 127.11.125.62:40427 requested a full tablet report, sending...
I20260925 02:24:56.630698 24640 ts_manager.cc:194] Registered new tserver with Master: 9c15470c987e43a6bc622ae93e69f2ab (127.11.125.3:36005)
I20260925 02:24:56.631209 24640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:35529
I20260925 02:24:56.634644 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:24:56.643499 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.635578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.11.125.62:40427)
0925 02:24:56.635701 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.635703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.635784 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:24:56.635857 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.635858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.635870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.635939 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:56.636624 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.637378 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.638437 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.638724 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.638793 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.640313 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:56.640315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:56.640315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:56.640323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:56.642088 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:56.642499 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:56.642500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:56.642500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:56.642585 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:56.642803 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:56.642803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:56.642804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:56.642930 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:56.643037 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.643196 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.643283 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260925 02:24:56.645725 24640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:24:56.647996 24640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:24:56.656744 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:36005 (local address 127.0.0.1:39488)
0925 02:24:56.651682 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.651736 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:56.651803 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:56.652366 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:56.652369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:56.652371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:56.652443 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.652446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.653792 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.653794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.655170 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.655173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.656406 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.656407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.656489 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.656506 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:24:56.656591 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:24:56.656648 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
I20260925 02:24:56.656738 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.11.125.3:36005)
0925 02:24:56.652169 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.652174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.652185 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:24:56.652219 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.652221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.652250 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.652328 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:56.652515 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.653652 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.655275 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.656318 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.656389 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.656448 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.656616 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.656655 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":62,"threads_started":1}
I20260925 02:24:56.657975 25042 tablet_service.cc:1467] Processing CreateTablet for tablet d37997e63f304205845f52a0b8c02e3a (DEFAULT_TABLE table=test-table [id=465d6b29ca164d71af30a38c5589add4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:56.658354 25042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37997e63f304205845f52a0b8c02e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:56.661319 25146 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Bootstrap starting.
I20260925 02:24:56.661410 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39929 (local address 127.0.0.1:60652)
0925 02:24:56.651912 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.651922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:56.651957 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:56.657372 (+ 5415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:56.657373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:56.657375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:56.657438 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.657440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.658626 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.658627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.660147 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.660150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.661137 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.661139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.661224 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.661229 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:24:56.661281 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:24:56.661329 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":80,"threads_started":1}
I20260925 02:24:56.661686 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.11.125.1:39929)
0925 02:24:56.657034 (+ 5410us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.657039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.657050 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:24:56.657087 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.657089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.657106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.657183 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:56.657569 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.658536 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.660268 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.661042 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.661133 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.661193 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.661565 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.661608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5372,"thread_start_us":51,"threads_started":1}
I20260925 02:24:56.662470 25146 tablet_bootstrap.cc:654] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:56.662391 24773 tablet_service.cc:1467] Processing CreateTablet for tablet d37997e63f304205845f52a0b8c02e3a (DEFAULT_TABLE table=test-table [id=465d6b29ca164d71af30a38c5589add4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:56.662731 24773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37997e63f304205845f52a0b8c02e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:56.662896 25146 log.cc:824] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:56.663637 25146 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: No bootstrap required, opened a new log
I20260925 02:24:56.663697 25146 ts_tablet_manager.cc:1397] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:24:56.664700 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:32841 (local address 127.0.0.1:53412)
0925 02:24:56.651704 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.651738 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:56.652058 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:56.661240 (+ 9182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:56.661241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:56.661243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:56.661287 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.661288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.662521 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.662522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.663549 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.663552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.664481 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.664482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.664558 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.664562 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:24:56.664594 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 02:24:56.664632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":107,"threads_started":1}
I20260925 02:24:56.665104 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.651391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.11.125.2:32841)
0925 02:24:56.661021 (+ 9630us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.661026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.661038 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:24:56.661081 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.661082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.661115 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.661362 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:56.661446 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.662418 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.663654 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.664373 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.664876 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.664957 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.664984 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.665021 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9589,"thread_start_us":47,"threads_started":1}
I20260925 02:24:56.665704 24908 tablet_service.cc:1467] Processing CreateTablet for tablet d37997e63f304205845f52a0b8c02e3a (DEFAULT_TABLE table=test-table [id=465d6b29ca164d71af30a38c5589add4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:24:56.666025 24908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37997e63f304205845f52a0b8c02e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:56.666064 25146 raft_consensus.cc:348] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.666226 25146 raft_consensus.cc:374] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:56.666285 25146 raft_consensus.cc:729] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c15470c987e43a6bc622ae93e69f2ab, State: Initialized, Role: FOLLOWER
I20260925 02:24:56.666749 25146 consensus_queue.cc:260] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.667052 25146 ts_tablet_manager.cc:1428] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:24:56.667160 25110 heartbeater.cc:503] Master 127.11.125.62:40427 was elected leader, sending a full tablet report...
I20260925 02:24:56.667533 25148 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: Bootstrap starting.
I20260925 02:24:56.668449 25148 tablet_bootstrap.cc:654] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:56.668758 25148 log.cc:824] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:56.669492 25151 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: Bootstrap starting.
I20260925 02:24:56.669543 25148 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: No bootstrap required, opened a new log
I20260925 02:24:56.669600 25148 ts_tablet_manager.cc:1397] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:24:56.670171 25151 tablet_bootstrap.cc:654] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:56.670853 25151 log.cc:824] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:56.672104 25148 raft_consensus.cc:348] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.672279 25148 raft_consensus.cc:374] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:56.672309 25148 raft_consensus.cc:729] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2bfaf621564fe7a8361b4aa7c9aafa, State: Initialized, Role: FOLLOWER
I20260925 02:24:56.672453 25148 consensus_queue.cc:260] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.672698 25148 ts_tablet_manager.cc:1428] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 02:24:56.672746 24840 heartbeater.cc:503] Master 127.11.125.62:40427 was elected leader, sending a full tablet report...
I20260925 02:24:56.673358 25151 tablet_bootstrap.cc:492] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: No bootstrap required, opened a new log
I20260925 02:24:56.673476 25151 ts_tablet_manager.cc:1397] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:24:56.675595 25151 raft_consensus.cc:348] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.675803 25151 raft_consensus.cc:374] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:56.675853 25151 raft_consensus.cc:729] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823580b626134903b28a98c6a9c38e01, State: Initialized, Role: FOLLOWER
I20260925 02:24:56.675976 25151 consensus_queue.cc:260] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.676215 25151 ts_tablet_manager.cc:1428] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:24:56.676266 24974 heartbeater.cc:503] Master 127.11.125.62:40427 was elected leader, sending a full tablet report...
I20260925 02:24:56.679792 25153 raft_consensus.cc:482] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:24:56.679905 25153 raft_consensus.cc:504] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.680543 25153 leader_election.cc:288] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c15470c987e43a6bc622ae93e69f2ab (127.11.125.3:36005), 823580b626134903b28a98c6a9c38e01 (127.11.125.2:32841)
I20260925 02:24:56.684407 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.680478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:42967 (local address 127.11.125.2:32841)
0925 02:24:56.680522 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.680524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.680881 (+ 357us) server_negotiation.cc:407] Connection header received
0925 02:24:56.680983 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.680984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.680995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.681060 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:56.681245 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.681990 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.683390 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.683976 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.684052 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.684062 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.684297 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.684337 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:24:56.684655 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.680414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:32841 (local address 127.11.125.1:42967)
0925 02:24:56.680781 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.680795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:56.680874 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:56.681064 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:56.681065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:56.681067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:56.681166 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.681168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.682092 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.682093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.683289 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.683292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.684124 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.684126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.684201 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.684206 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:24:56.684514 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 02:24:56.684561 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":98,"threads_started":1}
I20260925 02:24:56.685007 24926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37997e63f304205845f52a0b8c02e3a" candidate_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823580b626134903b28a98c6a9c38e01" is_pre_election: true
I20260925 02:24:56.685145 24926 raft_consensus.cc:2454] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2bfaf621564fe7a8361b4aa7c9aafa in term 0.
I20260925 02:24:56.685539 24727 leader_election.cc:302] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2bfaf621564fe7a8361b4aa7c9aafa, 823580b626134903b28a98c6a9c38e01; no voters:
I20260925 02:24:56.685654 25153 raft_consensus.cc:2781] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:24:56.685688 25153 raft_consensus.cc:482] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:24:56.685706 25153 raft_consensus.cc:3037] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:56.686614 25153 raft_consensus.cc:504] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.686730 25153 leader_election.cc:288] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [CANDIDATE]: Term 1 election: Requested vote from peers 9c15470c987e43a6bc622ae93e69f2ab (127.11.125.3:36005), 823580b626134903b28a98c6a9c38e01 (127.11.125.2:32841)
I20260925 02:24:56.687214 24926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37997e63f304205845f52a0b8c02e3a" candidate_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823580b626134903b28a98c6a9c38e01"
I20260925 02:24:56.687299 24926 raft_consensus.cc:3037] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:56.688078 24926 raft_consensus.cc:2454] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2bfaf621564fe7a8361b4aa7c9aafa in term 1.
I20260925 02:24:56.688171 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.680626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:53437 (local address 127.11.125.3:36005)
0925 02:24:56.680694 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.680697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.684803 (+ 4106us) server_negotiation.cc:407] Connection header received
0925 02:24:56.684939 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.684941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.684952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.684982 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:24:56.685152 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.685881 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.687218 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.687793 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.687844 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.687853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.688057 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.688103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:24:56.688377 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.680418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:36005 (local address 127.11.125.1:53437)
0925 02:24:56.684741 (+ 4323us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.684752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:56.684831 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:56.685047 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:56.685048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:56.685050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:24:56.685079 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.685080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.686003 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.686004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.687097 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:56.687099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.687915 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.687916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:56.687974 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.687977 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:24:56.688279 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 02:24:56.688317 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4311,"thread_start_us":95,"threads_started":1}
I20260925 02:24:56.688385 24727 leader_election.cc:302] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2bfaf621564fe7a8361b4aa7c9aafa, 823580b626134903b28a98c6a9c38e01; no voters:
I20260925 02:24:56.688570 25153 raft_consensus.cc:2781] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:24:56.688722 25153 raft_consensus.cc:686] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [term 1 LEADER]: Becoming Leader. State: Replica: 3c2bfaf621564fe7a8361b4aa7c9aafa, State: Running, Role: LEADER
I20260925 02:24:56.688824 25153 consensus_queue.cc:237] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } }
I20260925 02:24:56.688829 25061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37997e63f304205845f52a0b8c02e3a" candidate_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c15470c987e43a6bc622ae93e69f2ab" is_pre_election: true
I20260925 02:24:56.688975 25061 raft_consensus.cc:2454] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2bfaf621564fe7a8361b4aa7c9aafa in term 0.
I20260925 02:24:56.689101 25063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37997e63f304205845f52a0b8c02e3a" candidate_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c15470c987e43a6bc622ae93e69f2ab"
I20260925 02:24:56.689165 25063 raft_consensus.cc:3037] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:56.689641 24640 catalog_manager.cc:5495] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c2bfaf621564fe7a8361b4aa7c9aafa (127.11.125.1). New cstate: current_term: 1 leader_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c2bfaf621564fe7a8361b4aa7c9aafa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 39929 } health_report { overall_health: HEALTHY } } }
I20260925 02:24:56.690016 25063 raft_consensus.cc:2454] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2bfaf621564fe7a8361b4aa7c9aafa in term 1.
I20260925 02:24:56.706077 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.702523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.11.125.1:39929)
0925 02:24:56.702596 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:56.702598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:56.702608 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:24:56.702645 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:56.702646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:56.702667 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:56.702776 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:24:56.702930 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.703709 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.704811 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:56.705167 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:56.705244 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:56.705473 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:24:56.705631 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:24:56.705711 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:56.705973 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:24:56.705997 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:24:56.709685 24926 raft_consensus.cc:1261] T d37997e63f304205845f52a0b8c02e3a P 823580b626134903b28a98c6a9c38e01 [term 1 FOLLOWER]: Refusing update from remote peer 3c2bfaf621564fe7a8361b4aa7c9aafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:56.710201 25157 consensus_queue.cc:1035] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "823580b626134903b28a98c6a9c38e01" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:56.710981 25063 raft_consensus.cc:1261] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 1 FOLLOWER]: Refusing update from remote peer 3c2bfaf621564fe7a8361b4aa7c9aafa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:56.711453 25157 consensus_queue.cc:1035] T d37997e63f304205845f52a0b8c02e3a P 3c2bfaf621564fe7a8361b4aa7c9aafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c15470c987e43a6bc622ae93e69f2ab" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:56.714650 25160 mvcc.cc:205] Tried to move back new op lower bound from 7333081484119506944 to 7333081484038156288. Current Snapshot: MvccSnapshot[applied={T|T < 7333081484119506944}]
I20260925 02:24:56.715279 25163 mvcc.cc:205] Tried to move back new op lower bound from 7333081484119506944 to 7333081484038156288. Current Snapshot: MvccSnapshot[applied={T|T < 7333081484119506944}]
I20260925 02:24:56.717577 25161 mvcc.cc:205] Tried to move back new op lower bound from 7333081484119506944 to 7333081484038156288. Current Snapshot: MvccSnapshot[applied={T|T < 7333081484119506944}]
I20260925 02:24:56.719525 24640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:24:56.719623 24640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:24:56.720443 24640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6119ec7147db4279854f261d7d8f6161: Sending DeleteTablet for 3 replicas of tablet d37997e63f304205845f52a0b8c02e3a
I20260925 02:24:56.721087 24908 tablet_service.cc:1514] Processing DeleteTablet for tablet d37997e63f304205845f52a0b8c02e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:24:56 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:53412
I20260925 02:24:56.721144 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24679
I20260925 02:24:56.721099 25042 tablet_service.cc:1514] Processing DeleteTablet for tablet d37997e63f304205845f52a0b8c02e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:24:56 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:39488
I20260925 02:24:56.725132 25170 tablet_replica.cc:330] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: stopping tablet replica
I20260925 02:24:56.725323 25170 raft_consensus.cc:2229] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:24:56.725523 25170 raft_consensus.cc:2258] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:24:56.726119 25170 ts_tablet_manager.cc:1905] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:24:56.727774 24626 connection.cc:535] client connection to 127.11.125.1:39929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 02:24:56.727873 24626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:24:56.728000 25170 ts_tablet_manager.cc:1918] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:24:56.728072 25170 log.cc:1196] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/d37997e63f304205845f52a0b8c02e3a
I20260925 02:24:56.728435 25170 ts_tablet_manager.cc:1939] T d37997e63f304205845f52a0b8c02e3a P 9c15470c987e43a6bc622ae93e69f2ab: Deleting consensus metadata
I20260925 02:24:56.728616 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24844
I20260925 02:24:56.729017 24624 catalog_manager.cc:4841] TS 9c15470c987e43a6bc622ae93e69f2ab (127.11.125.3:36005): tablet d37997e63f304205845f52a0b8c02e3a (table test-table [id=465d6b29ca164d71af30a38c5589add4]) successfully deleted
I20260925 02:24:56.730110 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.729946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:39929 (local address 127.0.0.1:60670)
0925 02:24:56.730015 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.730071 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:39929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:24:56.730299 24626 catalog_manager.cc:4584] TS 3c2bfaf621564fe7a8361b4aa7c9aafa (127.11.125.1:39929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d37997e63f304205845f52a0b8c02e3a: Network error: Client connection negotiation failed: client connection to 127.11.125.1:39929: connect: Connection refused (error 111)
I20260925 02:24:56.734265 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24977
I20260925 02:24:56.734298 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:56.734185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:32841 (local address 127.0.0.1:53428)
0925 02:24:56.734238 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:56.734264 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.2:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:24:56.741191 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 24607
I20260925 02:24:56.748947 11764 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:24:57.093169 11764 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:24:57.093278 11764 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:24:57.437721 11764 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:24:57.437 UTC [23668] LOG: received smart shutdown request
2026-09-25 02:24:57.441 UTC [23668] LOG: background worker "logical replication launcher" (PID 23677) exited with exit code 1
2026-09-25 02:24:57.442 UTC [23672] LOG: shutting down
2026-09-25 02:24:57.614 UTC [23668] LOG: database system is shut down
2026-09-25T02:24:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49552 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:24:57 dist-test-slave-hj7s krb5kdc[25180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:24:57 dist-test-slave-hj7s krb5kdc[25180](info): set up 2 sockets
Sep 25 02:24:57 dist-test-slave-hj7s krb5kdc[25180](info): commencing operation
krb5kdc: starting...
W20260925 02:24:58.784394 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:24:58 dist-test-slave-hj7s krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:24:58.905200 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40341
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:24:58.985762 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:58.985970 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:58.985991 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:58.987123 25193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:58.987164 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:58.987179 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:58.987229 25193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:58.987247 25193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:58.988750 25193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:58.989030 25193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:58.989264 25193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:24:58.994307 25193 server_base.cc:1034] running on GCE node
W20260925 02:24:58.994277 25199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:58.994256 25200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:58.994277 25202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:58.994663 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:58.994925 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:58.996078 25193 hybrid_clock.cc:648] HybridClock initialized: now 1790303098996068 us; error 30 us; skew 500 ppm
Sep 25 02:24:58 dist-test-slave-hj7s krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:58.998539 25193 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:24:58.998981 25193 webserver.cc:460] Webserver started at http://127.11.125.62:36465/ using document root <none> and password file <none>
I20260925 02:24:58.999161 25193 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:58.999200 25193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:58.999285 25193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:24:59.000326 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "10136459610e49c69238f6f0f2eb3f52"
format_stamp: "Formatted at 2026-09-25 02:24:58 on dist-test-slave-hj7s"
server_key: "3b96ade68c57198f27ce2cafda5f287f"
server_key_iv: "170ff27960c18ad5ef2156945e1deccf"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.000707 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "10136459610e49c69238f6f0f2eb3f52"
format_stamp: "Formatted at 2026-09-25 02:24:58 on dist-test-slave-hj7s"
server_key: "3b96ade68c57198f27ce2cafda5f287f"
server_key_iv: "170ff27960c18ad5ef2156945e1deccf"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.002074 25193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:24:59.002822 25209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:59.002995 25193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:24:59.003108 25193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "10136459610e49c69238f6f0f2eb3f52"
format_stamp: "Formatted at 2026-09-25 02:24:58 on dist-test-slave-hj7s"
server_key: "3b96ade68c57198f27ce2cafda5f287f"
server_key_iv: "170ff27960c18ad5ef2156945e1deccf"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.003178 25193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:59.016322 25193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:59.018184 25193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:59.018347 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:59.022594 25193 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:40341
I20260925 02:24:59.022630 25262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:40341 every 8 connection(s)
I20260925 02:24:59.022964 25193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:24:59.023804 25263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:59.025486 25263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:24:59.029847 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25193
I20260925 02:24:59.029990 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:24:59.030220 11764 external_mini_cluster.cc:1428] Setting key 11bc87cca67d33a50de40685f0750255
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:59.035599 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.026302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.11.125.62:40341)
0925 02:24:59.026432 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.026436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.026463 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:24:59.026980 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.026982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.026999 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.027076 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.027622 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.028421 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.029449 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.029757 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.029836 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.032289 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.032296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.032297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.032304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.033946 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.034334 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.034337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.034338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.034434 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.034668 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.034669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.034669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.034883 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.034977 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.035204 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.035423 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260925 02:24:59.035602 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.026157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40341 (local address 127.0.0.1:42880)
0925 02:24:59.026308 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.026343 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.026851 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.027128 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.027131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.027383 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.027536 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.027540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.028507 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.028508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.029377 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.029380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.029850 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.029852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.029960 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.030629 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.030640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.032202 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.034040 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.034047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.034049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.034245 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.034504 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.034505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.034506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.034590 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.034955 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.034961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.035039 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.035134 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.035338 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260925 02:24:59.036649 25263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:24:59.037052 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.036946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37154)
0925 02:24:59.036981 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.037031 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 02:24:59.037184 25212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
I20260925 02:24:59.037642 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.037540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37164)
0925 02:24:59.037589 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.037612 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 02:24:59.037778 25263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:38335: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.11.125.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
I20260925 02:24:59.056629 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:38335
--webserver_interface=127.11.125.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:24:59.086992 25263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } attempt: 1
I20260925 02:24:59.087610 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.087468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37166)
0925 02:24:59.087533 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.087576 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:24:59.087970 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.087869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37174)
0925 02:24:59.087915 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.087943 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:24:59.088124 25263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:38335: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
W20260925 02:24:59.139411 25270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:59.139617 25270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:59.139640 25270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:59.140825 25270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:59.140869 25270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:59.140884 25270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:59.140897 25270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:59.140972 25270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:59.142503 25270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.125.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:59.142755 25270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:59.142987 25270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:59.147786 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:59.147786 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:59.147851 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:59.148145 25270 server_base.cc:1034] running on GCE node
I20260925 02:24:59.148396 25270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:59.148768 25270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:59.149971 25270 hybrid_clock.cc:648] HybridClock initialized: now 1790303099149929 us; error 34 us; skew 500 ppm
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:59.152510 25270 init.cc:377] Logged in from keytab as kudu/127.11.125.61@KRBTEST.COM (short username kudu)
I20260925 02:24:59.153136 25270 webserver.cc:460] Webserver started at http://127.11.125.61:43321/ using document root <none> and password file <none>
I20260925 02:24:59.153424 25270 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:59.153499 25270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:59.153653 25270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:24:59.154770 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data/instance:
uuid: "ebb9c29b8d8b42389fd905c5f50aa46b"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "a32a6ecb910893a5a283439a43b6ffe5"
server_key_iv: "e9f575f40050606a5d7cad22467fd0df"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.155082 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal/instance:
uuid: "ebb9c29b8d8b42389fd905c5f50aa46b"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "a32a6ecb910893a5a283439a43b6ffe5"
server_key_iv: "e9f575f40050606a5d7cad22467fd0df"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.156283 25263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } attempt: 2
I20260925 02:24:59.156355 25270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:24:59.156678 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.156550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37182)
0925 02:24:59.156613 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.156648 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 02:24:59.157100 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.156987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37184)
0925 02:24:59.157039 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.157072 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 02:24:59.157213 25263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:38335: Network error: Client connection negotiation failed: client connection to 127.11.125.61:38335: connect: Connection refused (error 111)
I20260925 02:24:59.157354 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:59.157527 25270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:24:59.157639 25270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "ebb9c29b8d8b42389fd905c5f50aa46b"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "a32a6ecb910893a5a283439a43b6ffe5"
server_key_iv: "e9f575f40050606a5d7cad22467fd0df"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.157728 25270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:59.173213 25270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:59.175143 25270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:59.175307 25270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:59.179602 25270 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.61:38335
I20260925 02:24:59.179634 25339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.61:38335 every 8 connection(s)
I20260925 02:24:59.180037 25270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
I20260925 02:24:59.181002 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:59.181627 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25270
I20260925 02:24:59.181727 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-1/wal/instance
I20260925 02:24:59.181946 11764 external_mini_cluster.cc:1428] Setting key 890044e1bb22b98f88a969b0699cd5cf
I20260925 02:24:59.182680 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.60@KRBTEST.COM" created.
I20260925 02:24:59.192376 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.183442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40341 (local address 127.0.0.1:42884)
0925 02:24:59.183628 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.183678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.184354 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.184607 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.184611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.184864 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.185113 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.185117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.185808 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.185809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.186599 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.186600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.187012 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.187013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.187138 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.187639 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.187648 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.189008 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.190850 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.190857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.190858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.191112 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.191315 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.191316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.191316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.191441 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.191895 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.191901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.192010 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.192097 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.192208 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 02:24:59.192451 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.183482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.11.125.62:40341)
0925 02:24:59.183536 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.183538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.183777 (+ 239us) server_negotiation.cc:407] Connection header received
0925 02:24:59.184471 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.184473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.184488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.184544 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.185208 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.185707 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.186692 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.186912 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.187001 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.189105 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.189107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.189107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.189115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.190766 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.191183 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.191184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.191184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.191250 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.191549 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.191550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.191550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.191816 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.191909 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.192171 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.192284 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:24:59.193053 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:24:59.201020 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.193311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37200)
0925 02:24:59.193353 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.193364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.193404 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.193708 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.193709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.193837 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.193891 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.193892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.195232 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.195233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.195825 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.195827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.196523 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.196524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.196581 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.196993 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.196998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.198360 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.200024 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.200025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.200026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.200177 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.200399 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.200400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.200401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.200469 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.200723 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.200724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.200757 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.200808 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.200892 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 02:24:59.201227 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.193381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.11.125.61:38335)
0925 02:24:59.193541 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.193547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.193563 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:24:59.193586 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.193589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.193602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.193678 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.193976 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.195120 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.195927 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.196435 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.196512 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.198445 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.198453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.198453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.198459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.199944 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.200251 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.200253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.200253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.200303 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.200527 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.200528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.200528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.200662 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.200728 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.200830 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.201087 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260925 02:24:59.201805 25340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:24:59.202191 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.202072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:59988)
0925 02:24:59.202124 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.202171 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:24:59.202337 25290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
I20260925 02:24:59.202754 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.202657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60000)
0925 02:24:59.202710 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.202736 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:24:59.202889 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:46545: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
I20260925 02:24:59.208537 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:46545
--webserver_interface=127.11.125.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:24:59.248391 25263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } attempt: 3
I20260925 02:24:59.252125 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } attempt: 1
I20260925 02:24:59.252542 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.252385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60008)
0925 02:24:59.252448 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.252493 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:24:59.253580 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.253472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60022)
0925 02:24:59.253523 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.253555 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:24:59.253666 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:46545: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
I20260925 02:24:59.256168 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.248876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.11.125.61:38335)
0925 02:24:59.248947 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.248950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.248962 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:24:59.249062 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.249063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.249073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.249139 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.249483 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.249998 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.250913 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.251294 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.251361 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.253619 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.253621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.253621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.253667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.255016 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.255367 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.255368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.255369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.255426 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.255644 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.255645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.255645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.255769 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.255842 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.256003 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.256081 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:24:59.256175 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.248781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37216)
0925 02:24:59.248853 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.248887 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.248977 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.249158 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.249160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.249327 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.249395 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.249398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.250097 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.250098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.250801 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.250805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.251404 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.251405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.251511 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.251992 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.251997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.253531 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.255105 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.255106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.255106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.255281 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.255513 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.255514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.255514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.255580 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.255844 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.255845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.255892 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.255977 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.256080 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:24:59.256630 25263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:24:59.257019 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.256873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60030)
0925 02:24:59.256954 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.256988 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:24:59.257355 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.257264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60046)
0925 02:24:59.257311 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.257335 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:24:59.257489 25263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:46545: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
I20260925 02:24:59.288647 25263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } attempt: 1
I20260925 02:24:59.289172 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.289039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60048)
0925 02:24:59.289097 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.289137 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:24:59.289597 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.289513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60052)
0925 02:24:59.289555 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.289580 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 02:24:59.289705 25263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:46545: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
W20260925 02:24:59.292310 25347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:24:59.292484 25347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:24:59.292506 25347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:24:59.293659 25347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:24:59.293699 25347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:24:59.293712 25347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:24:59.293725 25347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:24:59.293736 25347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:24:59.295161 25347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:40341,127.11.125.61:38335,127.11.125.60:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.125.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:24:59.295346 25347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:24:59.295508 25347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:24:59.300411 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:24:59.300450 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:59.300616 25347 server_base.cc:1034] running on GCE node
W20260925 02:24:59.300463 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:24:59.301059 25347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:24:59.301394 25347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:24:59.302536 25347 hybrid_clock.cc:648] HybridClock initialized: now 1790303099302526 us; error 27 us; skew 500 ppm
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:24:59.305094 25347 init.cc:377] Logged in from keytab as kudu/127.11.125.60@KRBTEST.COM (short username kudu)
I20260925 02:24:59.305490 25347 webserver.cc:460] Webserver started at http://127.11.125.60:43287/ using document root <none> and password file <none>
I20260925 02:24:59.305804 25347 fs_manager.cc:359] Metadata directory not provided
I20260925 02:24:59.305853 25347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:24:59.305943 25347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:24:59.307046 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data/instance:
uuid: "573b659aacd4480291f253d7d2dbbd82"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "48120d53c965cd0835f777e43336d1ad"
server_key_iv: "fc98350aba56c182f65208e01385c294"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.307343 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal/instance:
uuid: "573b659aacd4480291f253d7d2dbbd82"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "48120d53c965cd0835f777e43336d1ad"
server_key_iv: "fc98350aba56c182f65208e01385c294"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.308576 25347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:24:59.309319 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:59.309508 25347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:24:59.309616 25347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "573b659aacd4480291f253d7d2dbbd82"
format_stamp: "Formatted at 2026-09-25 02:24:59 on dist-test-slave-hj7s"
server_key: "48120d53c965cd0835f777e43336d1ad"
server_key_iv: "fc98350aba56c182f65208e01385c294"
server_key_version: "encryptionkey@0"
I20260925 02:24:59.309692 25347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:24:59.321866 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } attempt: 2
I20260925 02:24:59.322300 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.322167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60066)
0925 02:24:59.322233 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.322272 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:24:59.322690 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.322571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60068)
0925 02:24:59.322637 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.322666 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:24:59.322785 25340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:46545: Network error: Client connection negotiation failed: client connection to 127.11.125.60:46545: connect: Connection refused (error 111)
I20260925 02:24:59.338176 25347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:24:59.340085 25347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:24:59.340260 25347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:24:59.344790 25347 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.60:46545
I20260925 02:24:59.344854 25414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.60:46545 every 8 connection(s)
I20260925 02:24:59.345528 25347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
I20260925 02:24:59.346511 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:24:59.348217 25415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:24:59.353192 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25347
I20260925 02:24:59.353330 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790302942477976-11764-0/minicluster-data/master-2/wal/instance
I20260925 02:24:59.353536 11764 external_mini_cluster.cc:1428] Setting key 62382779e34fe7221fdd5dce191cfb87
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:24:59.357631 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.349020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.11.125.62:40341)
0925 02:24:59.349080 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.349082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.349281 (+ 199us) server_negotiation.cc:407] Connection header received
0925 02:24:59.349720 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.349721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.349730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.349785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.350354 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.350834 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.351773 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.351948 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.351999 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.354518 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.354519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.354519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.354526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.356096 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.356611 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.356612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.356613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.356665 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.356955 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.356956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.356957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.357070 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.357150 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.357432 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.357515 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:24:59.357764 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.348979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:40341 (local address 127.0.0.1:42898)
0925 02:24:59.349150 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.349193 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.349640 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.349850 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.349854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.350093 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.350285 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.350289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.350916 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.350917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.351700 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.351702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.352019 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.352020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.352125 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.352650 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.352666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.354425 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.356209 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.356217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.356219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.356535 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.356730 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.356730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.356731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.356863 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.357169 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.357178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.357296 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.357374 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.357489 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260925 02:24:59.358570 25415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:24:59.363255 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.353953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.11.125.60:46545)
0925 02:24:59.354082 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.354087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.354126 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:24:59.354195 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.354196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.354213 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.354275 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.354946 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.356123 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.357132 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.357535 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.357605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.360295 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.360303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.360303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.360309 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.361963 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.362369 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.362372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.362372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.362437 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.362631 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.362632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.362632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.362826 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.363000 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.363068 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.363142 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 02:24:59.364851 25263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } attempt: 2
I20260925 02:24:59.366176 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.358978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.11.125.61:38335)
0925 02:24:59.359032 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.359034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.359043 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:24:59.359070 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.359071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.359082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.359157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.359478 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.359968 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.360887 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.361209 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.361275 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.363361 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.363363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.363364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.363371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.364988 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.365359 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.365360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.365360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.365422 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.365616 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.365617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.365617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.365755 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.365814 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.365996 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.366072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:24:59.366425 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.358883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:38335 (local address 127.0.0.1:37226)
0925 02:24:59.358933 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.358944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.358981 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.359178 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.359178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.359335 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.359391 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.359392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.360044 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.360046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.360791 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.360793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.361295 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.361296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.361375 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.361759 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.361764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.363263 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.365081 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.365082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.365082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.365297 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.365482 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.365482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.365483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.365549 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.365843 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.365843 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.365885 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.366251 (+ 366us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.366338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:24:59.366958 25415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:24:59.373512 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.364224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.11.125.62:40341)
0925 02:24:59.364271 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.364273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.364297 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:24:59.364367 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.364367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.364377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.364431 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.364893 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.365612 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.366581 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.366890 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.366930 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.369470 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.369471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.369471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.369478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.371248 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.371681 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.371682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.371682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.371734 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.371963 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.371964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.371964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.372086 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.372716 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.372992 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.373069 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:24:59.373728 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.365191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.11.125.60:46545)
0925 02:24:59.365223 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.365225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.365235 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:24:59.365300 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.365300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.365311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.365391 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.365656 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.366102 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.367042 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.367370 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.367659 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.370032 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.370033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.370034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.370040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.372460 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.372959 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.372960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.372960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.373018 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.373232 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.373232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.373233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.373321 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.373408 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.373562 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.373648 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 02:24:59.373729 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.365159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60076)
0925 02:24:59.365203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.365213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.365245 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.365402 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.365404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.365557 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.365602 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.365604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.366189 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.366190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.366962 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.366965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.367465 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.367467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.367528 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.367887 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.367893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.369943 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.372548 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.372548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.372549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.372784 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.373099 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.373100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.373101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.373160 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.373402 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.373403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.373445 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.373536 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.373633 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:24:59.375460 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.367218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60084)
0925 02:24:59.367396 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.367406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.367448 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.367594 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.367595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.367752 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.367787 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.367789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.369116 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.369117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.369574 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.369576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.370108 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.370108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.370170 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.370770 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.370776 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.372249 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.374275 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.374276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.374277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.374496 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.374712 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.374713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.374713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.374767 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.375068 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.375068 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.375101 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.375203 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.375306 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 02:24:59.375572 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.367262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.11.125.60:46545)
0925 02:24:59.367457 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.367459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.367478 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:24:59.367496 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.367497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.367505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.367592 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.368491 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.369036 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.369708 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.370007 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.370070 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.372347 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.372348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.372348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.372356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.374197 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.374593 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.374594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.374594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.374651 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.374835 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.374836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.374836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.374992 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.375065 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.375233 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.375398 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
I20260925 02:24:59.375687 25263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: Bootstrap starting.
I20260925 02:24:59.376484 25263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:59.376825 25263 log.cc:824] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:59.377013 25415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: Bootstrap starting.
I20260925 02:24:59.377676 25263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: No bootstrap required, opened a new log
I20260925 02:24:59.377745 25415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:59.378048 25415 log.cc:824] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:59.378873 25415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: No bootstrap required, opened a new log
I20260925 02:24:59.379346 25263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.379510 25263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:59.379542 25263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10136459610e49c69238f6f0f2eb3f52, State: Initialized, Role: FOLLOWER
I20260925 02:24:59.379655 25263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.379946 25263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:59.381515 25415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
W20260925 02:24:59.381732 25435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 02:24:59.381747 25415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:59.381795 25415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573b659aacd4480291f253d7d2dbbd82, State: Initialized, Role: FOLLOWER
W20260925 02:24:59.381897 25435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:24:59.381963 25415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:24:59.382341 25415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:59.382609 25193 master_runner.cc:186] Error getting master registration for 127.11.125.61:38335: OK, instance_id { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" instance_seqno: 1790303099178390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260925 02:24:59.385049 25448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 02:24:59.385080 25193 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:24:59.385110 25448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:24:59.385500 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.385708 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.385918 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.386013 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.386703 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.374660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.11.125.61:38335)
0925 02:24:59.377009 (+ 2349us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.377011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.377019 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:24:59.377065 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.377066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.377093 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.377155 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.377797 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.378470 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.379435 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.379712 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.379790 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.382751 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.382752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.382753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.382760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.384576 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.384999 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.385000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.385000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.385058 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.385308 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.385309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.385309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.385441 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.386064 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.386508 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.386592 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2315}
I20260925 02:24:59.388049 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:24:59.387015 25347 master_runner.cc:186] Error getting master registration for 127.11.125.61:38335: OK, instance_id { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" instance_seqno: 1790303099178390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:24:59.389246 25347 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:24:59.397876 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.389756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.11.125.61:38335)
0925 02:24:59.389790 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.389792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.390021 (+ 229us) server_negotiation.cc:407] Connection header received
0925 02:24:59.390120 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.390121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.390141 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.390171 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.391009 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.391739 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.392792 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.393162 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.393227 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.394765 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.394766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.394767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.394773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.396500 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.396892 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.396893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.396894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.396980 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.397236 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.397237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.397237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.397386 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.397480 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.397707 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.397785 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:24:59.401820 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.389593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.11.125.62:40341)
0925 02:24:59.389639 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.389641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.390289 (+ 648us) server_negotiation.cc:407] Connection header received
0925 02:24:59.390398 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.390399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.390410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.390439 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.392051 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.392764 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.393822 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.394101 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.394146 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.398744 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.398745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.398746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.398754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.400530 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.400955 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.400956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.400957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.401012 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.401234 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.401235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.401235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.401363 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.401451 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.401655 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.401732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:24:59.402001 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.390034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.11.125.60:46545)
0925 02:24:59.390071 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.390073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.391162 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 02:24:59.391254 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.391255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.391265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.391294 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.394230 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.394953 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.396345 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.396682 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.396725 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.398773 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.398775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.398775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.398782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.400701 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.401131 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.401132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.401132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.401188 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.401394 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.401395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.401395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.401518 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.401614 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.401825 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.401910 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:24:59.404373 25424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:24:59.404469 25424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.404706 25424 leader_election.cc:288] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb9c29b8d8b42389fd905c5f50aa46b (127.11.125.61:38335), 573b659aacd4480291f253d7d2dbbd82 (127.11.125.60:46545)
I20260925 02:24:59.405089 25389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10136459610e49c69238f6f0f2eb3f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573b659aacd4480291f253d7d2dbbd82" is_pre_election: true
I20260925 02:24:59.405267 25389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10136459610e49c69238f6f0f2eb3f52 in term 0.
I20260925 02:24:59.405475 25214 leader_election.cc:302] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10136459610e49c69238f6f0f2eb3f52, 573b659aacd4480291f253d7d2dbbd82; no voters:
I20260925 02:24:59.405665 25424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:24:59.405732 25424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:24:59.405755 25424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:59.406760 25424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.406878 25424 leader_election.cc:288] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [CANDIDATE]: Term 1 election: Requested vote from peers ebb9c29b8d8b42389fd905c5f50aa46b (127.11.125.61:38335), 573b659aacd4480291f253d7d2dbbd82 (127.11.125.60:46545)
I20260925 02:24:59.407258 25389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10136459610e49c69238f6f0f2eb3f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573b659aacd4480291f253d7d2dbbd82"
I20260925 02:24:59.407349 25389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:59.408268 25389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10136459610e49c69238f6f0f2eb3f52 in term 1.
I20260925 02:24:59.408842 25214 leader_election.cc:302] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10136459610e49c69238f6f0f2eb3f52, 573b659aacd4480291f253d7d2dbbd82; no voters:
I20260925 02:24:59.408954 25424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:24:59.409117 25424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [term 1 LEADER]: Becoming Leader. State: Replica: 10136459610e49c69238f6f0f2eb3f52, State: Running, Role: LEADER
I20260925 02:24:59.409220 25424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.409615 25457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10136459610e49c69238f6f0f2eb3f52. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.409734 25457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:59.410074 25459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:24:59.410491 25459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:24:59.413120 25389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [term 1 FOLLOWER]: Refusing update from remote peer 10136459610e49c69238f6f0f2eb3f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:59.413475 25457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:59.414855 25340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } attempt: 3
I20260925 02:24:59.414853 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10136459610e49c69238f6f0f2eb3f52. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.414949 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.418794 25457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.418875 25457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:59.419399 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.419471 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.419728 25460 mvcc.cc:205] Tried to move back new op lower bound from 7333081495192223744 to 7333081495180718080. Current Snapshot: MvccSnapshot[applied={T|T < 7333081495192223744}]
Sep 25 02:24:59 dist-test-slave-hj7s krb5kdc[25180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303099, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:24:59.420395 25461 mvcc.cc:205] Tried to move back new op lower bound from 7333081495192223744 to 7333081495180718080. Current Snapshot: MvccSnapshot[applied={T|T < 7333081495192223744}]
I20260925 02:24:59.421574 25459 catalog_manager.cc:1347] Generated new cluster ID: acbbd4d34a594597833b1509d4308097
I20260925 02:24:59.421656 25459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:24:59.423703 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.415132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:46545 (local address 127.0.0.1:60104)
0925 02:24:59.415243 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:24:59.415254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:24:59.415292 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:24:59.415523 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:24:59.415525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:24:59.415698 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:24:59.415751 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.415753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.416669 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.416670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.417543 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:24:59.417546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.418025 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.418026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:24:59.418107 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.418594 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:24:59.418600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:24:59.420460 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:24:59.422471 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.422473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.422473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.422696 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.422956 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:24:59.422957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:24:59.422957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:24:59.423043 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:24:59.423389 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:24:59.423390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:24:59.423451 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 02:24:59.423512 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:24:59.423604 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 02:24:59.424007 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:24:59.415374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.11.125.60:46545)
0925 02:24:59.415411 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:24:59.415413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:24:59.415419 (+ 6us) server_negotiation.cc:407] Connection header received
0925 02:24:59.415439 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:24:59.415440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:24:59.415448 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:24:59.415499 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:24:59.415821 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.416569 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.417643 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:24:59.417940 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:24:59.418006 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:24:59.420576 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:24:59.420578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:24:59.420578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:24:59.420585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:24:59.422355 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.422796 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.422798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.422798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.422869 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:24:59.423136 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:24:59.423137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:24:59.423138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:24:59.423282 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:24:59.423341 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:24:59.423849 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 02:24:59.423924 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:24:59.425607 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: Bootstrap starting.
I20260925 02:24:59.426374 25340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: Neither blocks nor log segments found. Creating new log.
I20260925 02:24:59.426697 25340 log.cc:824] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:24:59.434239 25340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: No bootstrap required, opened a new log
I20260925 02:24:59.436432 25340 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.436589 25340 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:24:59.436622 25340 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb9c29b8d8b42389fd905c5f50aa46b, State: Initialized, Role: FOLLOWER
I20260925 02:24:59.436771 25340 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } }
I20260925 02:24:59.437127 25340 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:24:59.437177 25312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10136459610e49c69238f6f0f2eb3f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b"
I20260925 02:24:59.437467 25313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10136459610e49c69238f6f0f2eb3f52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" is_pre_election: true
I20260925 02:24:59.437892 25312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:24:59.438827 25312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10136459610e49c69238f6f0f2eb3f52 in term 1.
I20260925 02:24:59.439070 25311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [term 1 FOLLOWER]: Refusing update from remote peer 10136459610e49c69238f6f0f2eb3f52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:24:59.439564 25424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:24:59.439917 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.441964 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.441250 25478 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10136459610e49c69238f6f0f2eb3f52. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.442157 25478 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.441356 25481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.442279 25481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:24:59.450026 25477 catalog_manager.cc:1259] Loaded cluster ID: acbbd4d34a594597833b1509d4308097
I20260925 02:24:59.450165 25477 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: loading cluster ID for follower catalog manager: success
W20260925 02:24:59.450315 25477 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 02:24:59.450400 25477 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:24:59.467662 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10136459610e49c69238f6f0f2eb3f52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10136459610e49c69238f6f0f2eb3f52" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 40341 } } peers { permanent_uuid: "ebb9c29b8d8b42389fd905c5f50aa46b" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 38335 } } peers { permanent_uuid: "573b659aacd4480291f253d7d2dbbd82" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 46545 } } }
I20260925 02:24:59.467816 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52 [sys.catalog]: This master's current role is: LEADER
I20260925 02:24:59.471563 25459 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:24:59.472254 25459 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:24:59.486100 25459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10136459610e49c69238f6f0f2eb3f52: Generated new TSK 0
I20260925 02:24:59.486297 25459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:25:00.385556 25448 catalog_manager.cc:1259] Loaded cluster ID: acbbd4d34a594597833b1509d4308097
I20260925 02:25:00.385660 25448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: loading cluster ID for follower catalog manager: success
I20260925 02:25:00.386492 25448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: acquiring CA information for follower catalog manager: success
I20260925 02:25:00.386811 25448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 573b659aacd4480291f253d7d2dbbd82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:25:00.451581 25477 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: acquiring CA information for follower catalog manager: success
I20260925 02:25:00.452000 25477 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebb9c29b8d8b42389fd905c5f50aa46b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:25:00.545128 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25193
I20260925 02:25:00.550727 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25270
I20260925 02:25:00.555724 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25347
2026-09-25T02:25:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2819 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:25:00 dist-test-slave-hj7s krb5kdc[25492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:25:00 dist-test-slave-hj7s krb5kdc[25492](info): set up 2 sockets
Sep 25 02:25:00 dist-test-slave-hj7s krb5kdc[25492](info): commencing operation
krb5kdc: starting...
W20260925 02:25:01.663203 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:25:01 dist-test-slave-hj7s krb5kdc[25492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:25:01.783744 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:37827
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:01.865367 25506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:01.865621 25506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:01.865662 25506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:01.866827 25506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:01.866901 25506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:01.866930 25506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:01.866962 25506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:01.866986 25506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:01.868448 25506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:37827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:01.868729 25506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:01.868988 25506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:01.873906 25511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:01.873994 25512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:01.874161 25506 server_base.cc:1034] running on GCE node
W20260925 02:25:01.873970 25514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:01.874498 25506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:01.874810 25506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:01.875980 25506 hybrid_clock.cc:648] HybridClock initialized: now 1790303101875966 us; error 47 us; skew 500 ppm
Sep 25 02:25:01 dist-test-slave-hj7s krb5kdc[25492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303101, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:01.878604 25506 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:01.879112 25506 webserver.cc:460] Webserver started at http://127.11.125.62:36959/ using document root <none> and password file <none>
I20260925 02:25:01.879371 25506 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:01.879441 25506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:01.879594 25506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:01.880703 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "b007084075da4409962c7505c007194e"
format_stamp: "Formatted at 2026-09-25 02:25:01 on dist-test-slave-hj7s"
server_key: "d2c84185825ebdec851e4a68cb424b4e"
server_key_iv: "49e81862f31d77891296ca3076727a7a"
server_key_version: "encryptionkey@0"
I20260925 02:25:01.881100 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "b007084075da4409962c7505c007194e"
format_stamp: "Formatted at 2026-09-25 02:25:01 on dist-test-slave-hj7s"
server_key: "d2c84185825ebdec851e4a68cb424b4e"
server_key_iv: "49e81862f31d77891296ca3076727a7a"
server_key_version: "encryptionkey@0"
I20260925 02:25:01.882504 25506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:25:01.883443 25521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:01.883615 25506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:01.883714 25506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "b007084075da4409962c7505c007194e"
format_stamp: "Formatted at 2026-09-25 02:25:01 on dist-test-slave-hj7s"
server_key: "d2c84185825ebdec851e4a68cb424b4e"
server_key_iv: "49e81862f31d77891296ca3076727a7a"
server_key_version: "encryptionkey@0"
I20260925 02:25:01.883795 25506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:01.906754 25506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:01.908710 25506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:01.908823 25506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:01.912887 25506 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:37827
I20260925 02:25:01.912985 25574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:37827 every 8 connection(s)
I20260925 02:25:01.913344 25506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:01.914234 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:01.916603 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e: Bootstrap starting.
I20260925 02:25:01.917275 25575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:01.917582 25575 log.cc:824] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:01.918335 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e: No bootstrap required, opened a new log
I20260925 02:25:01.918576 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25506
I20260925 02:25:01.918727 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:25:01.918962 11764 external_mini_cluster.cc:1428] Setting key f8e26bafa87497c6af346042e1686164
I20260925 02:25:01.920804 25575 raft_consensus.cc:348] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } }
I20260925 02:25:01.921041 25575 raft_consensus.cc:374] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:01.921080 25575 raft_consensus.cc:729] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b007084075da4409962c7505c007194e, State: Initialized, Role: FOLLOWER
I20260925 02:25:01.921214 25575 consensus_queue.cc:260] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } }
I20260925 02:25:01.921303 25575 raft_consensus.cc:388] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:01.921342 25575 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:01.921403 25575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:01.921960 25575 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } }
I20260925 02:25:01.922097 25575 leader_election.cc:302] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b007084075da4409962c7505c007194e; no voters:
I20260925 02:25:01.922269 25575 leader_election.cc:288] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:25:01.922339 25580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:01.922505 25580 raft_consensus.cc:686] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [term 1 LEADER]: Becoming Leader. State: Replica: b007084075da4409962c7505c007194e, State: Running, Role: LEADER
I20260925 02:25:01.922547 25575 sys_catalog.cc:564] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:01.922603 25580 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } }
I20260925 02:25:01.923081 25581 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b007084075da4409962c7505c007194e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } } }
I20260925 02:25:01.923071 25580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b007084075da4409962c7505c007194e. Latest consensus state: current_term: 1 leader_uuid: "b007084075da4409962c7505c007194e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007084075da4409962c7505c007194e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 37827 } } }
I20260925 02:25:01.923189 25581 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:01.923194 25580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:01.923441 25589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:01.925006 25589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:25:01.926712 25589 catalog_manager.cc:1347] Generated new cluster ID: 4df2c401d18c4131b13a2b5b43e85d3b
I20260925 02:25:01.926772 25589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:25:01 dist-test-slave-hj7s krb5kdc[25492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:01.933084 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:01.919531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.11.125.62:37827)
0925 02:25:01.919782 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:01.919793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:01.919828 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:25:01.921046 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:01.921055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:01.921124 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:01.921284 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:01.922059 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:01.923531 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:01.924590 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:01.925258 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:01.925366 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:01.928389 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:01.928393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:01.928393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:01.928408 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:01.930556 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:01.930999 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:01.931003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:01.931003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:01.931123 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:01.931337 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:01.931338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:01.931338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:01.931687 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:01.931841 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:01.932660 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 02:25:01.932820 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260925 02:25:01.934379 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:25:01.938649 25589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:25:01.939373 25589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:25:01.943583 25589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b007084075da4409962c7505c007194e: Generated new TSK 0
I20260925 02:25:01.943765 25589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:25:01.988706 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25506
2026-09-25T02:25:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1430 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:25:02 dist-test-slave-hj7s krb5kdc[25602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:25:02 dist-test-slave-hj7s krb5kdc[25602](info): set up 2 sockets
Sep 25 02:25:02 dist-test-slave-hj7s krb5kdc[25602](info): commencing operation
krb5kdc: starting...
W20260925 02:25:03.027195 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:25:03.147737 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:03.229321 25615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:03.229521 25615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:03.229542 25615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:03.230691 25615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:03.230734 25615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:03.230748 25615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:03.230787 25615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:03.230813 25615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:03.232370 25615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:03.232618 25615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:03.232843 25615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:03.237790 25623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.238013 25620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.237915 25615 server_base.cc:1034] running on GCE node
W20260925 02:25:03.237836 25621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.238322 25615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:03.238636 25615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:03.239809 25615 hybrid_clock.cc:648] HybridClock initialized: now 1790303103239795 us; error 44 us; skew 500 ppm
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:03.242332 25615 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:03.242753 25615 webserver.cc:460] Webserver started at http://127.11.125.62:35685/ using document root <none> and password file <none>
I20260925 02:25:03.242977 25615 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:03.243046 25615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:03.243191 25615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:03.244251 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "23dc63af47f9428891cf1d3cc1f78e23"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "32d84a575635e0cb916a2d32154eb4f7"
server_key_iv: "5404a96609e6290f812fa5e3c18e15d4"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.244616 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "23dc63af47f9428891cf1d3cc1f78e23"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "32d84a575635e0cb916a2d32154eb4f7"
server_key_iv: "5404a96609e6290f812fa5e3c18e15d4"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.246037 25615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:25:03.246764 25630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.246938 25615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.247038 25615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "23dc63af47f9428891cf1d3cc1f78e23"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "32d84a575635e0cb916a2d32154eb4f7"
server_key_iv: "5404a96609e6290f812fa5e3c18e15d4"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.247119 25615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:03.271526 25615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:03.274011 25615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:03.274192 25615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:03.278434 25615 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44513
I20260925 02:25:03.278499 25683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44513 every 8 connection(s)
I20260925 02:25:03.278884 25615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:03.279999 25684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:03.282461 25684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap starting.
I20260925 02:25:03.282850 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25615
I20260925 02:25:03.282984 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:25:03.283178 11764 external_mini_cluster.cc:1428] Setting key 18f2607d7c1fcae1bb4007183f649edd
I20260925 02:25:03.283175 25684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:03.283483 25684 log.cc:824] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:03.284313 25684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: No bootstrap required, opened a new log
I20260925 02:25:03.286695 25684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:03.286875 25684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:03.286906 25684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Initialized, Role: FOLLOWER
I20260925 02:25:03.287046 25684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:03.287146 25684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:03.287189 25684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:03.287243 25684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:03.287823 25684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:03.287938 25684 leader_election.cc:302] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23dc63af47f9428891cf1d3cc1f78e23; no voters:
I20260925 02:25:03.288105 25684 leader_election.cc:288] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:25:03.288167 25689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:03.288323 25689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 LEADER]: Becoming Leader. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Running, Role: LEADER
I20260925 02:25:03.288343 25684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:03.288501 25689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:03.288872 25689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23dc63af47f9428891cf1d3cc1f78e23. Latest consensus state: current_term: 1 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:03.288998 25689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:03.288879 25691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:03.289290 25691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:03.289661 25696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:03.290179 25696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:03.292727 25696 catalog_manager.cc:1347] Generated new cluster ID: 0d2d0ec67a5044928ee587d70c1810c9
I20260925 02:25:03.292783 25696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:25:03.295091 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.283703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.11.125.62:44513)
0925 02:25:03.283914 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.283924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.283961 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:25:03.284575 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.284584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.284650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.284838 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:03.285526 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.286968 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.287840 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.288482 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.290527 (+ 2045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.291623 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:03.291632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:03.291632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:03.291643 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:03.293544 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.293903 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.293906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.293907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.294013 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.294226 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.294227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.294227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.294466 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:03.294631 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.294728 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.294878 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260925 02:25:03.310360 25696 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:25:03.310936 25696 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:25:03.315085 25696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Generated new TSK 0
I20260925 02:25:03.315282 25696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:25:03.351814 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:03.450644 25711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:03.450850 25711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:03.450873 25711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:03.452086 25711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:03.452131 25711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:03.452157 25711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:25:03.453672 25711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:03.453922 25711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:03.454155 25711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:03.459404 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.459443 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.459604 25711 server_base.cc:1034] running on GCE node
W20260925 02:25:03.459409 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.459935 25711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:03.460196 25711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:03.461390 25711 hybrid_clock.cc:648] HybridClock initialized: now 1790303103461383 us; error 38 us; skew 500 ppm
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:03.464265 25711 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:25:03.464704 25711 webserver.cc:460] Webserver started at http://127.11.125.1:37051/ using document root <none> and password file <none>
I20260925 02:25:03.465013 25711 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:03.465083 25711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:03.465222 25711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:03.466271 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "5a965e6137984fa188aa5c83218a5161"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "9704a32ee56e10c1c56f624797834c6d"
server_key_iv: "77cdb59824cd6ca0aa3c8ca75bb2e783"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.466657 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "5a965e6137984fa188aa5c83218a5161"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "9704a32ee56e10c1c56f624797834c6d"
server_key_iv: "77cdb59824cd6ca0aa3c8ca75bb2e783"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.467975 25711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:25:03.468684 25726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.468859 25711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:03.468979 25711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "5a965e6137984fa188aa5c83218a5161"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "9704a32ee56e10c1c56f624797834c6d"
server_key_iv: "77cdb59824cd6ca0aa3c8ca75bb2e783"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.469069 25711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:03.504276 25711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:03.506731 25711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:03.506908 25711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:03.507150 25711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:03.507545 25711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:03.507602 25711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.507643 25711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:03.507673 25711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.513782 25711 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:44775
I20260925 02:25:03.513827 25839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:44775 every 8 connection(s)
I20260925 02:25:03.514149 25711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:25:03.517750 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25711
I20260925 02:25:03.517848 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:25:03.518049 11764 external_mini_cluster.cc:1428] Setting key bd2e8904cf443aebef45486dbda96647
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:03.525278 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.514885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:58821 (local address 127.11.125.62:44513)
0925 02:25:03.514944 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.514946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.515140 (+ 194us) server_negotiation.cc:407] Connection header received
0925 02:25:03.515615 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.515616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.515626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.515686 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:03.517008 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.517599 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.518844 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.519050 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.519102 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.522048 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:03.522049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:03.522049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:03.522056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:03.523561 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.524156 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.524157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.524157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.524208 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.524524 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.524524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.524525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.524622 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:03.524673 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.525145 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.525214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:25:03.525588 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.514851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:58821)
0925 02:25:03.515012 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.515053 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.515537 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.515738 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.515741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.516562 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:03.516891 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.516922 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.517699 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.517701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.518758 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.518762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.519135 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.519137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.519298 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.519776 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:03.519786 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:03.521949 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:03.523667 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.523676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.523679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.524071 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.524284 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.524285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.524286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.524452 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.524697 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:03.524706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:03.524976 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.525115 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.525322 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260925 02:25:03.526199 25840 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:03.526293 25840 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:03.526480 25840 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:03.527000 25647 ts_manager.cc:194] Registered new tserver with Master: 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775)
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
I20260925 02:25:03.527606 25647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:58821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:25:03.545454 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:03.627434 25845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:03.627678 25845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:03.627717 25845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:03.628870 25845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:03.628959 25845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:03.629007 25845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:25:03.630924 25845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:03.631256 25845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:03.631556 25845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:03.636694 25850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.636688 25851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.636688 25853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.636924 25845 server_base.cc:1034] running on GCE node
I20260925 02:25:03.637236 25845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:03.637496 25845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:03.638681 25845 hybrid_clock.cc:648] HybridClock initialized: now 1790303103638668 us; error 26 us; skew 500 ppm
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:03.641193 25845 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:25:03.641625 25845 webserver.cc:460] Webserver started at http://127.11.125.2:44559/ using document root <none> and password file <none>
I20260925 02:25:03.641842 25845 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:03.641888 25845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:03.642046 25845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:03.643081 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "1b9cc7d135b943f2902efd44d933f55a"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "56527cfdcd07f7dd73be554d3f798fe7"
server_key_iv: "aa4e72227686a95899b2c4dfab7a5231"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.643431 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9cc7d135b943f2902efd44d933f55a"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "56527cfdcd07f7dd73be554d3f798fe7"
server_key_iv: "aa4e72227686a95899b2c4dfab7a5231"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.644783 25845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:25:03.645530 25860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.645762 25845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:03.645877 25845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1b9cc7d135b943f2902efd44d933f55a"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "56527cfdcd07f7dd73be554d3f798fe7"
server_key_iv: "aa4e72227686a95899b2c4dfab7a5231"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.645951 25845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:03.664726 25845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:03.666689 25845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:03.666854 25845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:03.667094 25845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:03.667410 25845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:03.667444 25845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.667465 25845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:03.667515 25845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.673692 25845 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:37609
I20260925 02:25:03.673753 25974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:37609 every 8 connection(s)
I20260925 02:25:03.674075 25845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:25:03.680166 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25845
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:03.680295 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:25:03.680517 11764 external_mini_cluster.cc:1428] Setting key 7c7856d7e72dddf759947f671553a5cd
I20260925 02:25:03.683812 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.674891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:45863 (local address 127.11.125.62:44513)
0925 02:25:03.674960 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.674962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.675178 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:25:03.675739 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.675741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.675754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.675806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:03.676760 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.677348 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.678343 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.678551 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.678630 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.680857 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:03.680859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:03.680859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:03.680868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:03.682332 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.682817 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.682818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.682818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.682896 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.683132 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.683133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.683134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.683278 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:03.683348 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.683588 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.683677 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:25:03.683821 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.674886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:45863)
0925 02:25:03.675051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.675087 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.675612 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.675877 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.675886 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.676452 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:03.676640 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.676645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.677446 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.677447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.678238 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.678240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.678636 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.678637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.678764 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.679278 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:03.679286 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:03.680737 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:03.682413 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.682422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.682424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.682743 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.682969 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.682969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.682969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.683071 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.683341 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:03.683345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:03.683457 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.683544 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.683663 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260925 02:25:03.684288 25975 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:03.684393 25975 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:03.684585 25975 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:03.685036 25647 ts_manager.cc:194] Registered new tserver with Master: 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
I20260925 02:25:03.685405 25647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:45863
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:25:03.707136 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:03.790323 25980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:03.790562 25980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:03.790599 25980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:03.791826 25980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:03.791890 25980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:03.791934 25980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:25:03.793427 25980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:03.793682 25980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:03.793906 25980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:03.798909 25988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.798928 25986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:03.798947 25985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:03.799170 25980 server_base.cc:1034] running on GCE node
I20260925 02:25:03.799347 25980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:03.799537 25980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:03.800678 25980 hybrid_clock.cc:648] HybridClock initialized: now 1790303103800664 us; error 28 us; skew 500 ppm
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:03.803210 25980 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:25:03.803632 25980 webserver.cc:460] Webserver started at http://127.11.125.3:41265/ using document root <none> and password file <none>
I20260925 02:25:03.803853 25980 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:03.803928 25980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:03.804085 25980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:03.805186 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "7f709c1ec1d269272d0c92414924b068"
server_key_iv: "805a40aca3ff89f64845bc6c56e9feb8"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.805537 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "7f709c1ec1d269272d0c92414924b068"
server_key_iv: "805a40aca3ff89f64845bc6c56e9feb8"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.806838 25980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:03.807564 25995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.807737 25980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:03.807843 25980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "7f709c1ec1d269272d0c92414924b068"
server_key_iv: "805a40aca3ff89f64845bc6c56e9feb8"
server_key_version: "encryptionkey@0"
I20260925 02:25:03.807922 25980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:03.836236 25980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:03.838264 25980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:03.838454 25980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:03.838717 25980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:03.839064 25980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:03.839120 25980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.839169 25980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:03.839201 25980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:03.845749 25980 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:39883
I20260925 02:25:03.845794 26111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:39883 every 8 connection(s)
I20260925 02:25:03.846145 25980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:25:03 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303103, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:03.852459 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 25980
I20260925 02:25:03.852569 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:25:03.852794 11764 external_mini_cluster.cc:1428] Setting key 555ab634ebf8430d0726b86b630e9a42
I20260925 02:25:03.855531 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.846839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:33211 (local address 127.11.125.62:44513)
0925 02:25:03.846909 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.846911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.847117 (+ 206us) server_negotiation.cc:407] Connection header received
0925 02:25:03.847620 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.847621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.847631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.847684 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:03.848682 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.849287 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.850264 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.850524 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.850582 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.852815 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:03.852817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:03.852817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:03.852826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:03.854368 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.854733 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.854734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.854734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.854784 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.854986 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.854986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.854987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.855105 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:03.855160 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.855403 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.855465 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:25:03.855808 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.846834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:33211)
0925 02:25:03.847008 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.847046 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.847526 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.847757 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.847760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.848391 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:03.848602 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.848606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.849366 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.849366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.850168 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.850170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.850606 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.850607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.850714 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.851281 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:03.851290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:03.852727 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:03.854441 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.854448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.854450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.854667 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.854825 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:03.854825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:03.854826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:03.854914 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:03.855166 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:03.855171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:03.855285 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.855370 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.855509 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260925 02:25:03.856292 26112 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:03.856410 26112 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:03.856591 26112 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:03.857018 25647 ts_manager.cc:194] Registered new tserver with Master: 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883)
I20260925 02:25:03.857398 25647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:33211
I20260925 02:25:03.864610 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:25:03.871973 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.865995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.11.125.62:44513)
0925 02:25:03.866056 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.866058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.866165 (+ 107us) server_negotiation.cc:407] Connection header received
0925 02:25:03.866203 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.866204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.866212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.866255 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:03.866766 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.867218 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.867990 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.868163 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.868212 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.869351 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:03.869352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:03.869352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:03.869358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:03.870700 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.871097 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.871098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.871099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.871166 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:03.871383 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:03.871384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:03.871385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:03.871540 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:03.871649 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.871779 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.871870 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:25:03.873574 25647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:25:03.875126 25647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:25:03.881666 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.11.125.3:39883)
0925 02:25:03.877895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.877899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.877907 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:25:03.877939 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.877941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.877954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.878038 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:03.878306 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879128 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.880542 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.881238 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.881326 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.881405 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.881536 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.881579 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260925 02:25:03.881659 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39883 (local address 127.0.0.1:53832)
0925 02:25:03.877602 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.877642 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.877714 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.878057 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.878060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.878062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:03.878136 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.878138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879238 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.879239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.880445 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.880447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.881338 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.881339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.881424 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.881439 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.881512 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.881575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260925 02:25:03.882114 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44775 (local address 127.0.0.1:46666)
0925 02:25:03.877703 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.877715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.877771 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.878458 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.878459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.878461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:03.878498 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.878499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879759 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.879760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.880803 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.880806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.881809 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.881811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.881916 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.881920 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.881984 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.882026 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260925 02:25:03.882160 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.11.125.1:44775)
0925 02:25:03.878195 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.878199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.878209 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:25:03.878250 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.878252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.878268 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.878342 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:03.878591 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879660 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.880924 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.881656 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.881719 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.881770 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.882018 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.882062 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":77,"threads_started":1}
I20260925 02:25:03.882639 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37609 (local address 127.0.0.1:38872)
0925 02:25:03.877938 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:03.877945 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:03.878014 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:03.878257 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:03.878258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:03.878260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:03.878317 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.878319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879656 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.879657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.880667 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:03.880668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.882400 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.882401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:03.882450 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.882454 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:03.882531 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:25:03.882573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":99,"threads_started":1}
I20260925 02:25:03.882673 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:03.877689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.11.125.2:37609)
0925 02:25:03.877899 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:03.877903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:03.878011 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:25:03.878084 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:03.878086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:03.878100 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:03.878176 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:03.878403 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.879542 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.880746 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:03.881548 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:03.881626 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:03.881688 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:03.882551 (+ 863us) server_negotiation.cc:299] Negotiation successful
0925 02:25:03.882595 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
I20260925 02:25:03.882592 26044 tablet_service.cc:1467] Processing CreateTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 (DEFAULT_TABLE table=test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:03.883069 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bf3dcfc7a84b0fa91d73768c844099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:03.883702 25908 tablet_service.cc:1467] Processing CreateTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 (DEFAULT_TABLE table=test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:03.884044 25908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bf3dcfc7a84b0fa91d73768c844099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:03.885743 25774 tablet_service.cc:1467] Processing CreateTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 (DEFAULT_TABLE table=test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:03.886159 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bf3dcfc7a84b0fa91d73768c844099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:03.887017 26130 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Bootstrap starting.
I20260925 02:25:03.887799 26130 tablet_bootstrap.cc:654] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:03.888057 26131 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Bootstrap starting.
I20260925 02:25:03.888540 26130 log.cc:824] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:03.888689 26131 tablet_bootstrap.cc:654] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:03.889454 26131 log.cc:824] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:03.889730 26130 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: No bootstrap required, opened a new log
I20260925 02:25:03.889827 26130 ts_tablet_manager.cc:1397] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:25:03.890066 26133 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Bootstrap starting.
I20260925 02:25:03.890507 26131 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: No bootstrap required, opened a new log
I20260925 02:25:03.890602 26131 ts_tablet_manager.cc:1397] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:25:03.890771 26133 tablet_bootstrap.cc:654] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:03.891345 26133 log.cc:824] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:03.892271 26133 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: No bootstrap required, opened a new log
I20260925 02:25:03.892347 26133 ts_tablet_manager.cc:1397] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:03.892271 26130 raft_consensus.cc:348] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.892462 26130 raft_consensus.cc:374] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:03.892506 26130 raft_consensus.cc:729] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fddafc6e7ce4eb385f3ca85fd642cd2, State: Initialized, Role: FOLLOWER
I20260925 02:25:03.892629 26130 consensus_queue.cc:260] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.892668 26131 raft_consensus.cc:348] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.892952 26130 ts_tablet_manager.cc:1428] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:25:03.892988 26131 raft_consensus.cc:374] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:03.893049 26131 raft_consensus.cc:729] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9cc7d135b943f2902efd44d933f55a, State: Initialized, Role: FOLLOWER
I20260925 02:25:03.893147 26112 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
I20260925 02:25:03.893200 26131 consensus_queue.cc:260] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.893540 26131 ts_tablet_manager.cc:1428] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:25:03.893579 25975 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
I20260925 02:25:03.894942 26133 raft_consensus.cc:348] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.895102 26133 raft_consensus.cc:374] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:03.895145 26133 raft_consensus.cc:729] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a965e6137984fa188aa5c83218a5161, State: Initialized, Role: FOLLOWER
I20260925 02:25:03.895268 26133 consensus_queue.cc:260] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:03.895618 26133 ts_tablet_manager.cc:1428] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:25:03.895642 25840 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
W20260925 02:25:03.924772 25976 tablet.cc:2369] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:25:04.014889 25841 tablet.cc:2369] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:25:04.077993 26138 raft_consensus.cc:482] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:04.078137 26138 raft_consensus.cc:504] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:04.078593 26138 leader_election.cc:288] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
I20260925 02:25:04.082970 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.078950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:41091 (local address 127.11.125.3:39883)
0925 02:25:04.079000 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.079002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.079164 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:25:04.079197 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.079199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.079208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.079265 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:04.079510 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.080267 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.081967 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.082564 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.082643 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.082653 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.082844 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.082896 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 02:25:04.082970 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.078917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:34255 (local address 127.11.125.2:37609)
0925 02:25:04.078988 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.078991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.079099 (+ 108us) server_negotiation.cc:407] Connection header received
0925 02:25:04.079185 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.079186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.079197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.079265 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:04.079510 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.080270 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.081967 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.082560 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.082644 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.082653 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.082844 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.082896 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:25:04.082985 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.078829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39883 (local address 127.11.125.1:41091)
0925 02:25:04.079078 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.079089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.079147 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.079295 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.079296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.079301 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:04.079420 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.079424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.080382 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.080383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.081871 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.081873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.082660 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.082661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.082748 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.082757 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.082814 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.082870 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":109,"thread_start_us":78,"threads_started":1}
I20260925 02:25:04.082984 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.078810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37609 (local address 127.11.125.1:34255)
0925 02:25:04.078987 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.079008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.079105 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.079295 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.079296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.079298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:04.079428 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.079430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.080388 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.080388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.081866 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.081870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.082662 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.082662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.082748 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.082755 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.082814 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.082869 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":102,"thread_start_us":84,"threads_started":1}
I20260925 02:25:04.083438 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bf3dcfc7a84b0fa91d73768c844099" candidate_uuid: "5a965e6137984fa188aa5c83218a5161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" is_pre_election: true
I20260925 02:25:04.083617 26065 raft_consensus.cc:2454] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a965e6137984fa188aa5c83218a5161 in term 0.
I20260925 02:25:04.083668 25928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bf3dcfc7a84b0fa91d73768c844099" candidate_uuid: "5a965e6137984fa188aa5c83218a5161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9cc7d135b943f2902efd44d933f55a" is_pre_election: true
I20260925 02:25:04.083814 25928 raft_consensus.cc:2454] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a965e6137984fa188aa5c83218a5161 in term 0.
I20260925 02:25:04.083902 25728 leader_election.cc:302] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161, 8fddafc6e7ce4eb385f3ca85fd642cd2; no voters:
I20260925 02:25:04.084036 26138 raft_consensus.cc:2781] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:25:04.084115 26138 raft_consensus.cc:482] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:25:04.084146 26138 raft_consensus.cc:3037] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:04.084856 26138 raft_consensus.cc:504] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:04.085041 26138 leader_election.cc:288] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 1 election: Requested vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
I20260925 02:25:04.085206 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bf3dcfc7a84b0fa91d73768c844099" candidate_uuid: "5a965e6137984fa188aa5c83218a5161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
I20260925 02:25:04.085227 25928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bf3dcfc7a84b0fa91d73768c844099" candidate_uuid: "5a965e6137984fa188aa5c83218a5161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9cc7d135b943f2902efd44d933f55a"
I20260925 02:25:04.085295 26065 raft_consensus.cc:3037] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:04.085294 25928 raft_consensus.cc:3037] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:04.086225 26065 raft_consensus.cc:2454] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a965e6137984fa188aa5c83218a5161 in term 1.
I20260925 02:25:04.086225 25928 raft_consensus.cc:2454] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a965e6137984fa188aa5c83218a5161 in term 1.
I20260925 02:25:04.086397 25728 leader_election.cc:302] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161, 8fddafc6e7ce4eb385f3ca85fd642cd2; no voters:
I20260925 02:25:04.086539 26138 raft_consensus.cc:2781] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:04.086730 26138 raft_consensus.cc:686] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 LEADER]: Becoming Leader. State: Replica: 5a965e6137984fa188aa5c83218a5161, State: Running, Role: LEADER
I20260925 02:25:04.086833 26138 consensus_queue.cc:237] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:04.087728 25647 catalog_manager.cc:5495] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a965e6137984fa188aa5c83218a5161 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "5a965e6137984fa188aa5c83218a5161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } health_report { overall_health: UNKNOWN } } }
W20260925 02:25:04.096692 26113 tablet.cc:2369] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:25:04.111742 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.109453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.11.125.1:44775)
0925 02:25:04.109526 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.109528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.109550 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:25:04.109589 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.109590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.109610 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.109667 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:04.109785 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.110248 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.111072 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.111259 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.111314 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.111456 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:04.111564 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:25:04.111627 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.111661 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.111679 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:25:04.113822 26065 raft_consensus.cc:1261] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Refusing update from remote peer 5a965e6137984fa188aa5c83218a5161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:04.113910 25928 raft_consensus.cc:1261] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Refusing update from remote peer 5a965e6137984fa188aa5c83218a5161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:04.114203 26138 consensus_queue.cc:1035] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:04.114370 26141 consensus_queue.cc:1035] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:04.117044 26145 mvcc.cc:205] Tried to move back new op lower bound from 7333081514447032320 to 7333081514340241408. Current Snapshot: MvccSnapshot[applied={T|T < 7333081514447032320}]
I20260925 02:25:04.117746 26146 mvcc.cc:205] Tried to move back new op lower bound from 7333081514447032320 to 7333081514340241408. Current Snapshot: MvccSnapshot[applied={T|T < 7333081514447032320}]
I20260925 02:25:04.119191 25646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:04.119285 25646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:04.120113 25646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Sending DeleteTablet for 3 replicas of tablet f7bf3dcfc7a84b0fa91d73768c844099
I20260925 02:25:04.120464 25774 tablet_service.cc:1514] Processing DeleteTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:04 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46666
I20260925 02:25:04.120558 26044 tablet_service.cc:1514] Processing DeleteTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:04 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:53832
I20260925 02:25:04.120590 25908 tablet_service.cc:1514] Processing DeleteTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:04 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38872
I20260925 02:25:04.120818 26157 tablet_replica.cc:330] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: stopping tablet replica
I20260925 02:25:04.120832 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25711
I20260925 02:25:04.120822 26155 tablet_replica.cc:330] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: stopping tablet replica
I20260925 02:25:04.120992 26157 raft_consensus.cc:2229] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:04.121119 26157 raft_consensus.cc:2258] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:04.122068 26157 ts_tablet_manager.cc:1905] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:04.123569 26157 ts_tablet_manager.cc:1918] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:04.123633 26157 log.cc:1196] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/f7bf3dcfc7a84b0fa91d73768c844099
I20260925 02:25:04.123839 26157 ts_tablet_manager.cc:1939] T f7bf3dcfc7a84b0fa91d73768c844099 P 1b9cc7d135b943f2902efd44d933f55a: Deleting consensus metadata
I20260925 02:25:04.124199 25634 catalog_manager.cc:4841] TS 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): tablet f7bf3dcfc7a84b0fa91d73768c844099 (table test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]) successfully deleted
I20260925 02:25:04.124253 26156 tablet_replica.cc:330] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: stopping tablet replica
I20260925 02:25:04.124377 26156 raft_consensus.cc:2229] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:04.124507 26156 raft_consensus.cc:2258] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:04.124943 26156 ts_tablet_manager.cc:1905] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:04.126204 26156 ts_tablet_manager.cc:1918] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:04.126295 26156 log.cc:1196] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/f7bf3dcfc7a84b0fa91d73768c844099
I20260925 02:25:04.126539 26156 ts_tablet_manager.cc:1939] T f7bf3dcfc7a84b0fa91d73768c844099 P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting consensus metadata
I20260925 02:25:04.126922 25632 catalog_manager.cc:4841] TS 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): tablet f7bf3dcfc7a84b0fa91d73768c844099 (table test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]) successfully deleted
W20260925 02:25:04.128057 25633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:44775 (error 108)
I20260925 02:25:04.128255 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25845
I20260925 02:25:04.129292 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.129156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44775 (local address 127.0.0.1:46676)
0925 02:25:04.129216 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.129259 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 02:25:04.129434 25633 catalog_manager.cc:4584] TS 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7bf3dcfc7a84b0fa91d73768c844099: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44775: connect: Connection refused (error 111)
I20260925 02:25:04.133955 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25980
I20260925 02:25:04.139985 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 25615
I20260925 02:25:04.145606 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--webserver_interface=127.11.125.62
--webserver_port=35685
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:04.227584 26159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:04.227849 26159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:04.227900 26159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:04.229092 26159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:04.229161 26159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:04.229184 26159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:04.229206 26159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:04.229236 26159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:04.230695 26159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:04.230974 26159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:04.231207 26159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:04.240347 26168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.240340 26166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.240337 26165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:04.240552 26159 server_base.cc:1034] running on GCE node
I20260925 02:25:04.240876 26159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:04.241288 26159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:04.242467 26159 hybrid_clock.cc:648] HybridClock initialized: now 1790303104242463 us; error 36 us; skew 500 ppm
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:04.244995 26159 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:04.245434 26159 webserver.cc:460] Webserver started at http://127.11.125.62:35685/ using document root <none> and password file <none>
I20260925 02:25:04.245672 26159 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:04.245751 26159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:04.247143 26159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:04.247794 26175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.247959 26159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:04.248066 26159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "23dc63af47f9428891cf1d3cc1f78e23"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "32d84a575635e0cb916a2d32154eb4f7"
server_key_iv: "5404a96609e6290f812fa5e3c18e15d4"
server_key_version: "encryptionkey@0"
I20260925 02:25:04.248484 26159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:04.283954 26159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:04.286434 26159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:04.286607 26159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:04.290958 26159 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44513
I20260925 02:25:04.290997 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44513 every 8 connection(s)
I20260925 02:25:04.291412 26159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:04.293203 26229 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:25:04.293905 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap starting.
I20260925 02:25:04.301154 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26159
I20260925 02:25:04.301537 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:44775
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=37051
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:25:04.303411 26229 log.cc:824] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:04.305546 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:04.305867 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap complete.
I20260925 02:25:04.308152 26229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:04.308383 26229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Initialized, Role: FOLLOWER
I20260925 02:25:04.308559 26229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:04.308622 26229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:04.308674 26229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:04.308725 26229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:25:04.309816 26229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:04.309909 26229 leader_election.cc:302] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23dc63af47f9428891cf1d3cc1f78e23; no voters:
I20260925 02:25:04.310127 26229 leader_election.cc:288] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:25:04.310164 26233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:25:04.310276 26233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 LEADER]: Becoming Leader. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Running, Role: LEADER
I20260925 02:25:04.310421 26229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:04.310439 26233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:04.310743 26233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23dc63af47f9428891cf1d3cc1f78e23. Latest consensus state: current_term: 2 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:04.310750 26234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:04.310825 26233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:04.310870 26234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:04.311035 26239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:04.311890 26239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:25:04.312003 26239 catalog_manager.cc:1259] Loaded cluster ID: 0d2d0ec67a5044928ee587d70c1810c9
I20260925 02:25:04.312040 26239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:25:04.312273 26239 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:25:04.317040 26239 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:25:04.319484 26239 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:25:04.319681 26239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Loaded TSK: 0
I20260925 02:25:04.319921 26239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:25:04.387336 26231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:04.387559 26231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:04.387600 26231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:04.388741 26231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:04.388803 26231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:04.388860 26231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:25:04.390379 26231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:44775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=37051
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:04.390658 26231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:04.390889 26231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:04.397018 26259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.397032 26257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.397034 26256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:04.397303 26231 server_base.cc:1034] running on GCE node
I20260925 02:25:04.397492 26231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:04.397753 26231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:04.398910 26231 hybrid_clock.cc:648] HybridClock initialized: now 1790303104398885 us; error 41 us; skew 500 ppm
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:04.401495 26231 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:25:04.402004 26231 webserver.cc:460] Webserver started at http://127.11.125.1:37051/ using document root <none> and password file <none>
I20260925 02:25:04.402238 26231 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:04.402316 26231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:04.403674 26231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:04.404405 26266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.404583 26231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:04.404670 26231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "5a965e6137984fa188aa5c83218a5161"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "9704a32ee56e10c1c56f624797834c6d"
server_key_iv: "77cdb59824cd6ca0aa3c8ca75bb2e783"
server_key_version: "encryptionkey@0"
I20260925 02:25:04.405079 26231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:04.432096 26231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:04.434347 26231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:04.434502 26231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:04.434795 26231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:04.435336 26273 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:25:04.436817 26231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:25:04.436877 26231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:25:04.436975 26231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:25:04.437659 26231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:25:04.437702 26231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:04.437752 26273 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Bootstrap starting.
I20260925 02:25:04.445583 26231 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:44775
I20260925 02:25:04.445659 26382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:44775 every 8 connection(s)
I20260925 02:25:04.446113 26231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:25:04.447276 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26231
I20260925 02:25:04.447580 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:37609
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=44559
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:25:04.451436 26273 log.cc:824] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:04.454617 26273 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:04.455000 26273 tablet_bootstrap.cc:492] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Bootstrap complete.
I20260925 02:25:04.456372 26273 ts_tablet_manager.cc:1397] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.012s
I20260925 02:25:04.457760 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.446993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:46613 (local address 127.11.125.62:44513)
0925 02:25:04.447242 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.447250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.447292 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:25:04.448013 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.448027 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.448090 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.448228 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:04.448897 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.449813 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.450881 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.451515 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.451621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.454104 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:04.454107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:04.454108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:04.454120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:04.455860 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.456318 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.456321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.456322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.456425 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.456712 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.456714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.456714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.457028 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:04.457125 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.457445 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.457578 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260925 02:25:04.457760 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.446893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:46613)
0925 02:25:04.447108 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.447152 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.447772 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.448274 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.448279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.448578 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:04.448766 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.448771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.449965 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.449966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.450762 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.450764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.451729 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.451730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.451884 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.452503 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:04.452515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:04.453978 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:04.455974 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.455981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.455983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.456214 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.456525 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.456526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.456526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.456640 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.457105 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:04.457112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:04.457260 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.457373 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.457559 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":131,"threads_started":1}
I20260925 02:25:04.458149 26273 raft_consensus.cc:348] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:04.458554 26273 raft_consensus.cc:729] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a965e6137984fa188aa5c83218a5161, State: Initialized, Role: FOLLOWER
I20260925 02:25:04.458714 26273 consensus_queue.cc:260] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } }
I20260925 02:25:04.458819 26383 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:04.458913 26383 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:04.459040 26273 ts_tablet_manager.cc:1428] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260925 02:25:04.459111 26383 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:04.459776 26192 ts_manager.cc:194] Registered new tserver with Master: 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775)
I20260925 02:25:04.460886 26192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:46613
I20260925 02:25:04.462013 26383 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:25:04.468793 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.460550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.11.125.1:44775)
0925 02:25:04.461031 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.461037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.461048 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:25:04.461081 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.461083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.461099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.461175 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:04.461678 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.462652 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.463436 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.463758 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.463841 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.465979 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:04.465987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:04.465987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:04.465994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:04.467480 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.467892 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.467895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.467895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.467959 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.468182 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.468183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.468184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.468328 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:04.468416 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.468604 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.468698 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":70,"threads_started":1}
I20260925 02:25:04.468794 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.460359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44775 (local address 127.0.0.1:46684)
0925 02:25:04.460528 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.460570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.460640 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.461190 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.461197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.461524 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:04.461584 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.461587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.462745 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.462746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.463333 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.463335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.463841 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.463842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.463928 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.464426 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:04.464431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:04.465891 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:04.467576 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.467583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.467585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.467790 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.468034 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.468035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.468035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.468110 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.468409 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:04.468416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:04.468468 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.468568 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.468700 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 02:25:04.469247 26316 tablet_service.cc:1514] Processing DeleteTablet for tablet f7bf3dcfc7a84b0fa91d73768c844099 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:04 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46684
I20260925 02:25:04.469813 26392 tablet_replica.cc:330] stopping tablet replica
I20260925 02:25:04.469924 26392 raft_consensus.cc:2229] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:04.469988 26392 raft_consensus.cc:2258] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:04.470302 26392 ts_tablet_manager.cc:1905] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:04.472669 26392 ts_tablet_manager.cc:1918] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:04.472766 26392 log.cc:1196] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/f7bf3dcfc7a84b0fa91d73768c844099
I20260925 02:25:04.473033 26392 ts_tablet_manager.cc:1939] T f7bf3dcfc7a84b0fa91d73768c844099 P 5a965e6137984fa188aa5c83218a5161: Deleting consensus metadata
I20260925 02:25:04.473548 26178 catalog_manager.cc:4841] TS 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775): tablet f7bf3dcfc7a84b0fa91d73768c844099 (table test-table [id=6443fdbc6be64911a92b53d9d2e61f7e]) successfully deleted
W20260925 02:25:04.535887 26387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:04.536091 26387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:04.536120 26387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:04.537350 26387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:04.537393 26387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:04.537420 26387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:25:04.538921 26387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:37609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=44559
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:04.539155 26387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:04.539371 26387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:04.544796 26400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.544879 26398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.544770 26397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:04.545076 26387 server_base.cc:1034] running on GCE node
I20260925 02:25:04.545302 26387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:04.545564 26387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:04.546752 26387 hybrid_clock.cc:648] HybridClock initialized: now 1790303104546745 us; error 35 us; skew 500 ppm
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:04.549409 26387 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:25:04.549794 26387 webserver.cc:460] Webserver started at http://127.11.125.2:44559/ using document root <none> and password file <none>
I20260925 02:25:04.549976 26387 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:04.550027 26387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:04.551393 26387 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:25:04.552043 26407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.552224 26387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:04.552322 26387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1b9cc7d135b943f2902efd44d933f55a"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "56527cfdcd07f7dd73be554d3f798fe7"
server_key_iv: "aa4e72227686a95899b2c4dfab7a5231"
server_key_version: "encryptionkey@0"
I20260925 02:25:04.552668 26387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:04.563714 26387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:04.565737 26387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:04.565894 26387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:04.566138 26387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:04.566459 26387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:04.566493 26387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.566515 26387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:04.566560 26387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.572306 26387 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:37609
I20260925 02:25:04.572397 26523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:37609 every 8 connection(s)
I20260925 02:25:04.572721 26387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:25:04.574487 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26387
I20260925 02:25:04.574860 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:39883
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=41265
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:04.582773 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.573514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:42943 (local address 127.11.125.62:44513)
0925 02:25:04.573577 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.573579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.573791 (+ 212us) server_negotiation.cc:407] Connection header received
0925 02:25:04.574277 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.574278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.574292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.574337 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:04.575680 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.576277 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.577400 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.577632 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.577720 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.579870 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:04.579871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:04.579871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:04.579879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:04.581351 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.581768 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.581769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.581769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.581831 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.582101 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.582102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.582102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.582220 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:04.582293 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.582584 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.582683 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:25:04.582939 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.573512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:42943)
0925 02:25:04.573665 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.573700 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.574196 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.574430 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.574435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.575276 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:04.575578 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.575585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.576385 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.576387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.577308 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.577311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.577734 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.577735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.577855 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.578378 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:04.578386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:04.579766 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:04.581442 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.581449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.581451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.581675 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.581909 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.581910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.581910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.582025 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.582298 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:04.582306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:04.582445 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.582550 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.582702 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260925 02:25:04.583449 26524 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:04.583544 26524 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:04.583725 26524 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:04.584164 26192 ts_manager.cc:194] Registered new tserver with Master: 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
I20260925 02:25:04.584738 26192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:42943
W20260925 02:25:04.658056 26527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:04.658247 26527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:04.658269 26527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:04.659317 26527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:04.659350 26527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:04.659375 26527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:25:04.660808 26527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:39883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=41265
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:04.661062 26527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:04.661252 26527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:04.666115 26532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.666157 26533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:04.666126 26535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:04.666455 26527 server_base.cc:1034] running on GCE node
I20260925 02:25:04.666635 26527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:04.666821 26527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:04.667954 26527 hybrid_clock.cc:648] HybridClock initialized: now 1790303104667947 us; error 29 us; skew 500 ppm
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:04.670570 26527 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:25:04.670991 26527 webserver.cc:460] Webserver started at http://127.11.125.3:41265/ using document root <none> and password file <none>
I20260925 02:25:04.671222 26527 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:04.671298 26527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:04.672617 26527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:04.673235 26542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.673413 26527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.673516 26527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "7f709c1ec1d269272d0c92414924b068"
server_key_iv: "805a40aca3ff89f64845bc6c56e9feb8"
server_key_version: "encryptionkey@0"
I20260925 02:25:04.673859 26527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:04.683426 26527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:04.685743 26527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:04.685907 26527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:04.686134 26527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:04.686443 26527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:04.686499 26527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.686542 26527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:04.686570 26527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:04.692348 26527 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:39883
I20260925 02:25:04.692392 26657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:39883 every 8 connection(s)
I20260925 02:25:04.692745 26527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:25:04 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303104, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:04.700338 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26527
I20260925 02:25:04.702239 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.693734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:39885 (local address 127.11.125.62:44513)
0925 02:25:04.693818 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.693820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.693982 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:25:04.694473 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.694474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.694484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.694537 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:04.695433 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.695913 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.697012 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.697202 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.697260 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.699383 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:04.699384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:04.699384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:04.699391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:04.700629 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.701186 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.701187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.701188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.701254 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.701558 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.701559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.701559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.701669 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:04.701718 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.702058 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.702142 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:25:04.702324 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.693684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:39885)
0925 02:25:04.693852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.693892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.694388 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.694589 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.694592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.695143 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:04.695337 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.695342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.695994 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.695994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.696863 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.696866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.697278 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.697278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.697391 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.697910 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:04.697919 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:04.699286 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:04.700712 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.700721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.700723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.701113 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.701331 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:04.701333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:04.701333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:04.701480 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:04.701738 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:04.701746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:04.701920 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.702015 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.702134 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260925 02:25:04.702817 26658 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44513
I20260925 02:25:04.702927 26658 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:04.703115 26658 heartbeater.cc:511] Master 127.11.125.62:44513 requested a full tablet report, sending...
I20260925 02:25:04.703498 26192 ts_manager.cc:194] Registered new tserver with Master: 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883)
I20260925 02:25:04.703853 26192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:39885
I20260925 02:25:04.707513 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.700726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.11.125.62:44513)
0925 02:25:04.700849 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.700851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.701021 (+ 170us) server_negotiation.cc:407] Connection header received
0925 02:25:04.701083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.701084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.701094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.701166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:04.701870 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.702567 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.703529 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.703771 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.703855 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.705226 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:04.705227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:04.705227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:04.705235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:04.706337 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.706675 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.706676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.706677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.706741 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:04.706945 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:04.706946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:04.706946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:04.707083 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:04.707272 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.707312 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.707395 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260925 02:25:04.708390 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:25:04.711310 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.708722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.11.125.62:44513)
0925 02:25:04.708783 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.708785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.708921 (+ 136us) server_negotiation.cc:407] Connection header received
0925 02:25:04.709014 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.709015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.709056 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.709127 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:04.709239 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.709692 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.710569 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.710776 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.710850 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.711051 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:04.711143 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:25:04.711210 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.711246 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.711265 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:25:04.711628 26192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:25:04.713335 26192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:25:04.715626 26316 tablet_service.cc:1467] Processing CreateTablet for tablet 349e921cf8374f04bd98bc57021b7d8a (DEFAULT_TABLE table=test-table [id=e32f54efefa644d893b6789765acd186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:04.715807 26316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e921cf8374f04bd98bc57021b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:04.717293 26273 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Bootstrap starting.
I20260925 02:25:04.717644 26273 tablet_bootstrap.cc:654] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:04.719061 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.715438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.11.125.3:39883)
0925 02:25:04.715595 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.715599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.715610 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:25:04.715653 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.715655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.715667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.715760 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:04.715870 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.716713 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.718016 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.718659 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.718719 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.718759 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.718941 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.718982 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 02:25:04.719071 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.715207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39883 (local address 127.0.0.1:53838)
0925 02:25:04.715282 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.715296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.715338 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.715766 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.715767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.715769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:04.715811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.715813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.716822 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.716823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.717890 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.717893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.718738 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.718739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.718830 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.718855 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.718914 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.718970 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:25:04.719309 26273 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: No bootstrap required, opened a new log
I20260925 02:25:04.719359 26273 ts_tablet_manager.cc:1397] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:25:04.719699 26273 raft_consensus.cc:348] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.719779 26273 raft_consensus.cc:374] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:04.719821 26273 raft_consensus.cc:729] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a965e6137984fa188aa5c83218a5161, State: Initialized, Role: FOLLOWER
I20260925 02:25:04.719878 26273 consensus_queue.cc:260] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.719754 26591 tablet_service.cc:1467] Processing CreateTablet for tablet 349e921cf8374f04bd98bc57021b7d8a (DEFAULT_TABLE table=test-table [id=e32f54efefa644d893b6789765acd186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:04.719960 26273 ts_tablet_manager.cc:1428] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:04.720135 26591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e921cf8374f04bd98bc57021b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:04.721172 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.715285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37609 (local address 127.0.0.1:38886)
0925 02:25:04.715477 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:04.715489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:04.715540 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:04.717758 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:04.717759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:04.717761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:04.717807 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.717809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.718858 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.718859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.719980 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:04.719984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.720872 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.720873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:04.720975 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.720980 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:25:04.721040 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:25:04.721085 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":144,"threads_started":1}
I20260925 02:25:04.721187 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:04.717371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.11.125.2:37609)
0925 02:25:04.717576 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:04.717581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:04.717592 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:25:04.717631 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:04.717633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:04.717648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:04.717728 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:04.717889 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.718750 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.720059 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:04.720783 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:04.720874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:04.720953 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:04.721067 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:25:04.721107 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":120,"threads_started":1}
I20260925 02:25:04.722066 26456 tablet_service.cc:1467] Processing CreateTablet for tablet 349e921cf8374f04bd98bc57021b7d8a (DEFAULT_TABLE table=test-table [id=e32f54efefa644d893b6789765acd186]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:04.722388 26456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349e921cf8374f04bd98bc57021b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:04.723342 26668 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Bootstrap starting.
I20260925 02:25:04.724052 26668 tablet_bootstrap.cc:654] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:04.724858 26668 log.cc:824] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:04.725764 26668 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: No bootstrap required, opened a new log
I20260925 02:25:04.725842 26668 ts_tablet_manager.cc:1397] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:25:04.726078 26670 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Bootstrap starting.
I20260925 02:25:04.726635 26670 tablet_bootstrap.cc:654] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:04.727093 26670 log.cc:824] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:04.727500 26668 raft_consensus.cc:348] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.727680 26668 raft_consensus.cc:374] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:04.727728 26668 raft_consensus.cc:729] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fddafc6e7ce4eb385f3ca85fd642cd2, State: Initialized, Role: FOLLOWER
I20260925 02:25:04.727882 26670 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: No bootstrap required, opened a new log
I20260925 02:25:04.727846 26668 consensus_queue.cc:260] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.727938 26670 ts_tablet_manager.cc:1397] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:25:04.728156 26668 ts_tablet_manager.cc:1428] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:25:04.728173 26658 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
I20260925 02:25:04.729566 26670 raft_consensus.cc:348] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.729713 26670 raft_consensus.cc:374] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:04.729760 26670 raft_consensus.cc:729] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9cc7d135b943f2902efd44d933f55a, State: Initialized, Role: FOLLOWER
I20260925 02:25:04.729907 26670 consensus_queue.cc:260] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:04.730175 26670 ts_tablet_manager.cc:1428] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:25:04.730199 26524 heartbeater.cc:503] Master 127.11.125.62:44513 was elected leader, sending a full tablet report...
W20260925 02:25:04.823351 26525 tablet.cc:2369] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:25:04.943463 26659 tablet.cc:2369] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:25:04.946879 26384 tablet.cc:2369] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:25:05.010972 26673 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:05.011106 26673 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:05.011509 26673 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775)
I20260925 02:25:05.014560 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.011773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:33171 (local address 127.11.125.3:39883)
0925 02:25:05.011894 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:05.011897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:05.011935 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:25:05.012061 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:05.012062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:05.012071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:05.012117 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:05.012278 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.012750 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.013799 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.014188 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.014278 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:05.014287 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:05.014450 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 02:25:05.014486 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:25:05.014569 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.011800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:39465 (local address 127.11.125.1:44775)
0925 02:25:05.011953 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:05.011954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:05.011961 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:25:05.011985 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:05.011986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:05.012009 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:05.012095 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:05.012264 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.012754 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.013795 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.014213 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.014304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:05.014388 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:05.014454 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:25:05.014499 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260925 02:25:05.014568 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.011701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44775 (local address 127.11.125.2:39465)
0925 02:25:05.011776 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.011789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:05.011836 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:05.012100 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:05.012101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:05.012104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:05.012186 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:05.012188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.012847 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.012849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:05.013691 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:05.013694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.014299 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.014300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:05.014384 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:05.014387 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:05.014437 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 02:25:05.014477 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":53}
I20260925 02:25:05.014581 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.011716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39883 (local address 127.11.125.2:33171)
0925 02:25:05.011871 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.011884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:05.011952 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:05.012120 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:05.012121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:05.012122 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:05.012203 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:05.012205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.012839 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.012840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:05.013694 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:05.013696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.014278 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.014279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:05.014355 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:05.014371 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:25:05.014442 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:25:05.014483 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260925 02:25:05.014964 26337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e921cf8374f04bd98bc57021b7d8a" candidate_uuid: "1b9cc7d135b943f2902efd44d933f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a965e6137984fa188aa5c83218a5161" is_pre_election: true
I20260925 02:25:05.015143 26337 raft_consensus.cc:2454] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9cc7d135b943f2902efd44d933f55a in term 0.
I20260925 02:25:05.015280 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e921cf8374f04bd98bc57021b7d8a" candidate_uuid: "1b9cc7d135b943f2902efd44d933f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" is_pre_election: true
I20260925 02:25:05.015419 26410 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9cc7d135b943f2902efd44d933f55a, 5a965e6137984fa188aa5c83218a5161; no voters:
I20260925 02:25:05.015467 26611 raft_consensus.cc:2454] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b9cc7d135b943f2902efd44d933f55a in term 0.
I20260925 02:25:05.015568 26673 raft_consensus.cc:2781] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:25:05.015637 26673 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:25:05.015673 26673 raft_consensus.cc:3037] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:05.016454 26673 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:05.016607 26673 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [CANDIDATE]: Term 1 election: Requested vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775)
I20260925 02:25:05.016767 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e921cf8374f04bd98bc57021b7d8a" candidate_uuid: "1b9cc7d135b943f2902efd44d933f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
I20260925 02:25:05.016855 26611 raft_consensus.cc:3037] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:05.016829 26337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349e921cf8374f04bd98bc57021b7d8a" candidate_uuid: "1b9cc7d135b943f2902efd44d933f55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a965e6137984fa188aa5c83218a5161"
I20260925 02:25:05.016966 26337 raft_consensus.cc:3037] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:05.017616 26611 raft_consensus.cc:2454] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9cc7d135b943f2902efd44d933f55a in term 1.
I20260925 02:25:05.017725 26337 raft_consensus.cc:2454] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b9cc7d135b943f2902efd44d933f55a in term 1.
I20260925 02:25:05.017849 26409 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2; no voters:
I20260925 02:25:05.017942 26673 raft_consensus.cc:2781] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:05.018132 26673 raft_consensus.cc:686] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 1 LEADER]: Becoming Leader. State: Replica: 1b9cc7d135b943f2902efd44d933f55a, State: Running, Role: LEADER
I20260925 02:25:05.018236 26673 consensus_queue.cc:237] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:05.019080 26192 catalog_manager.cc:5495] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2). New cstate: current_term: 1 leader_uuid: "1b9cc7d135b943f2902efd44d933f55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } health_report { overall_health: UNKNOWN } } }
I20260925 02:25:05.072654 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.070551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.11.125.2:37609)
0925 02:25:05.070596 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:05.070598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:05.070605 (+ 7us) server_negotiation.cc:407] Connection header received
0925 02:25:05.070669 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:05.070670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:05.070688 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:05.070745 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:05.070826 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.071284 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.071959 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:05.072143 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:05.072194 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:05.072317 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:05.072415 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:25:05.072479 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:05.072511 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 02:25:05.072528 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:25:05.074579 26611 raft_consensus.cc:1261] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Refusing update from remote peer 1b9cc7d135b943f2902efd44d933f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:05.074584 26337 raft_consensus.cc:1261] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Refusing update from remote peer 1b9cc7d135b943f2902efd44d933f55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:05.074985 26676 consensus_queue.cc:1035] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:05.075129 26673 consensus_queue.cc:1035] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:05.077895 26679 mvcc.cc:205] Tried to move back new op lower bound from 7333081518382379008 to 7333081518155268096. Current Snapshot: MvccSnapshot[applied={T|T < 7333081518382379008 or (T in {7333081518382379008})}]
I20260925 02:25:05.078006 26680 mvcc.cc:205] Tried to move back new op lower bound from 7333081518382379008 to 7333081518155268096. Current Snapshot: MvccSnapshot[applied={T|T < 7333081518382379008 or (T in {7333081518382379008})}]
I20260925 02:25:05.079430 26191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:05.079543 26191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:05.080371 26191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Sending DeleteTablet for 3 replicas of tablet 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:05.080744 26591 tablet_service.cc:1514] Processing DeleteTablet for tablet 349e921cf8374f04bd98bc57021b7d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:05 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:53838
I20260925 02:25:05.080787 26316 tablet_service.cc:1514] Processing DeleteTablet for tablet 349e921cf8374f04bd98bc57021b7d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:05 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46684
I20260925 02:25:05.080770 26456 tablet_service.cc:1514] Processing DeleteTablet for tablet 349e921cf8374f04bd98bc57021b7d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:05 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:38886
I20260925 02:25:05.081018 26690 tablet_replica.cc:330] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: stopping tablet replica
I20260925 02:25:05.081049 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26231
I20260925 02:25:05.081127 26690 raft_consensus.cc:2229] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:25:05.081220 26690 raft_consensus.cc:2258] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:05.081732 26690 ts_tablet_manager.cc:1905] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:05.083962 26690 ts_tablet_manager.cc:1918] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:05.084052 26690 log.cc:1196] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:05.084373 26690 ts_tablet_manager.cc:1939] T 349e921cf8374f04bd98bc57021b7d8a P 1b9cc7d135b943f2902efd44d933f55a: Deleting consensus metadata
I20260925 02:25:05.084859 26179 catalog_manager.cc:4841] TS 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): tablet 349e921cf8374f04bd98bc57021b7d8a (table test-table [id=e32f54efefa644d893b6789765acd186]) successfully deleted
I20260925 02:25:05.087175 26688 tablet_replica.cc:330] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: stopping tablet replica
I20260925 02:25:05.087309 26688 raft_consensus.cc:2229] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:05.087433 26688 raft_consensus.cc:2258] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:05.087853 26688 ts_tablet_manager.cc:1905] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:05.089356 26688 ts_tablet_manager.cc:1918] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:05.089449 26688 log.cc:1196] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:05.089708 26688 ts_tablet_manager.cc:1939] T 349e921cf8374f04bd98bc57021b7d8a P 8fddafc6e7ce4eb385f3ca85fd642cd2: Deleting consensus metadata
I20260925 02:25:05.090150 26177 catalog_manager.cc:4841] TS 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): tablet 349e921cf8374f04bd98bc57021b7d8a (table test-table [id=e32f54efefa644d893b6789765acd186]) successfully deleted
W20260925 02:25:05.092147 26178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:44775 (error 108)
I20260925 02:25:05.092324 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26387
I20260925 02:25:05.093106 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.092650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:44775 (local address 127.0.0.1:46692)
0925 02:25:05.093007 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.093063 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
W20260925 02:25:05.093834 26178 catalog_manager.cc:4584] TS 5a965e6137984fa188aa5c83218a5161 (127.11.125.1:44775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 349e921cf8374f04bd98bc57021b7d8a: Network error: Client connection negotiation failed: client connection to 127.11.125.1:44775: connect: Connection refused (error 111)
I20260925 02:25:05.097769 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26527
I20260925 02:25:05.102761 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26159
I20260925 02:25:05.108587 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--webserver_interface=127.11.125.62
--webserver_port=35685
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:05.190217 26692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:05.190455 26692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:05.190493 26692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:05.191674 26692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:05.191742 26692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:05.191766 26692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:05.191787 26692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:05.191817 26692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:05.193401 26692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:05.193671 26692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:05.193904 26692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:05.199172 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:05.199190 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.199383 26692 server_base.cc:1034] running on GCE node
W20260925 02:25:05.199150 26700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.199628 26692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:05.199923 26692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:05.201086 26692 hybrid_clock.cc:648] HybridClock initialized: now 1790303105201062 us; error 37 us; skew 500 ppm
Sep 25 02:25:05 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:05.203637 26692 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:05.204169 26692 webserver.cc:460] Webserver started at http://127.11.125.62:35685/ using document root <none> and password file <none>
I20260925 02:25:05.204407 26692 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:05.204476 26692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:05.205853 26692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.206499 26707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.206740 26692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.206849 26692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "23dc63af47f9428891cf1d3cc1f78e23"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "32d84a575635e0cb916a2d32154eb4f7"
server_key_iv: "5404a96609e6290f812fa5e3c18e15d4"
server_key_version: "encryptionkey@0"
I20260925 02:25:05.207183 26692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:05.215324 26692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:05.217810 26692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:05.217976 26692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:05.222059 26692 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44513
I20260925 02:25:05.222088 26759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44513 every 8 connection(s)
I20260925 02:25:05.222467 26692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:05.223839 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26692
I20260925 02:25:05.224126 26760 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:25:05.224166 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:44775
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=37051
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:25:05.225251 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap starting.
I20260925 02:25:05.234462 26760 log.cc:824] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:05.236806 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:05.237154 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23: Bootstrap complete.
I20260925 02:25:05.239455 26760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:05.239673 26760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Initialized, Role: FOLLOWER
I20260925 02:25:05.239849 26760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:05.239913 26760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:05.239960 26760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:05.240011 26760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:25:05.241082 26760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:05.241174 26760 leader_election.cc:302] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23dc63af47f9428891cf1d3cc1f78e23; no voters:
I20260925 02:25:05.241402 26760 leader_election.cc:288] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:25:05.241441 26764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:25:05.241564 26764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [term 3 LEADER]: Becoming Leader. State: Replica: 23dc63af47f9428891cf1d3cc1f78e23, State: Running, Role: LEADER
I20260925 02:25:05.241657 26764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } }
I20260925 02:25:05.241685 26760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:05.242131 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23dc63af47f9428891cf1d3cc1f78e23. Latest consensus state: current_term: 3 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:05.242136 26765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23dc63af47f9428891cf1d3cc1f78e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23dc63af47f9428891cf1d3cc1f78e23" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44513 } } }
I20260925 02:25:05.242218 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:05.242265 26765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23dc63af47f9428891cf1d3cc1f78e23 [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:05.242470 26769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:05.243350 26769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:25:05.243453 26769 catalog_manager.cc:1259] Loaded cluster ID: 0d2d0ec67a5044928ee587d70c1810c9
I20260925 02:25:05.243691 26769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:25:05.243978 26769 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd2e5fca9ad google::LogMessage::Fail() at ??:0
@ 0x7fd2e5fcd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd2e5fca4de google::LogMessage::Flush() at ??:0
@ 0x7fd2e5fcba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd2edeeb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd2edef48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2e7e980d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2e7e8d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2e8cf16db start_thread at ??:0
@ 0x7fd2e4ac861f clone at ??:0
W20260925 02:25:05.343497 26762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:05.343797 26762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:05.343837 26762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:05.345548 26762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:05.345616 26762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:05.345661 26762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:25:05.347899 26762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:44775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=37051
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:05.348165 26762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:05.348430 26762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:05.353896 26796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:05.353904 26797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.354149 26762 server_base.cc:1034] running on GCE node
W20260925 02:25:05.354223 26799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.354532 26762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:05.354830 26762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:05.356030 26762 hybrid_clock.cc:648] HybridClock initialized: now 1790303105356008 us; error 48 us; skew 500 ppm
Sep 25 02:25:05 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:05.359225 26762 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:25:05.359786 26762 webserver.cc:460] Webserver started at http://127.11.125.1:37051/ using document root <none> and password file <none>
I20260925 02:25:05.360086 26762 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:05.360172 26762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:05.362025 26762 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.362883 26806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.363089 26762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.363194 26762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "5a965e6137984fa188aa5c83218a5161"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "9704a32ee56e10c1c56f624797834c6d"
server_key_iv: "77cdb59824cd6ca0aa3c8ca75bb2e783"
server_key_version: "encryptionkey@0"
I20260925 02:25:05.363665 26762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:05.375223 26762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:05.377775 26762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:05.377949 26762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:05.378245 26762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:05.378789 26813 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:25:05.380239 26762 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:25:05.380306 26762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:25:05.380347 26762 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:25:05.381330 26762 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:25:05.381381 26762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.381458 26813 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Bootstrap starting.
I20260925 02:25:05.390024 26762 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:44775
I20260925 02:25:05.390180 26922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:44775 every 8 connection(s)
I20260925 02:25:05.390529 26762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:25:05.391999 26813 log.cc:824] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:05.394275 26813 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:05.394639 26813 tablet_bootstrap.cc:492] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Bootstrap complete.
I20260925 02:25:05.395612 26813 ts_tablet_manager.cc:1397] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260925 02:25:05.396646 26813 raft_consensus.cc:348] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:05.396896 26813 raft_consensus.cc:729] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a965e6137984fa188aa5c83218a5161, State: Initialized, Role: FOLLOWER
I20260925 02:25:05.397114 26813 consensus_queue.cc:260] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:05.397399 26813 ts_tablet_manager.cc:1428] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 02:25:05.400202 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26762
I20260925 02:25:05.400544 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:37609
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=44559
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:05.499061 26925 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:25:05.391273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:45815)
0925 02:25:05.391528 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.391581 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:05.392245 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:05.498834 (+106589us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
W20260925 02:25:05.499276 26923 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44513 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:44513: Client connection negotiation failed: client connection to 127.11.125.62:44513: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:25:05.499660 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.499509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:37333)
0925 02:25:05.499569 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.499629 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:25:05.500022 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.499920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:56801)
0925 02:25:05.499971 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.500001 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:25:05.500155 26923 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:25:05.513278 26928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:05.513465 26928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:05.513489 26928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:05.514624 26928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:05.514664 26928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:05.514771 26928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:25:05.516220 26928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:37609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=44559
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:05.516455 26928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:05.516682 26928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:05.521777 26934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:05.521812 26937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:05.522543 26933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.522807 26928 server_base.cc:1034] running on GCE node
I20260925 02:25:05.523022 26928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:05.523217 26928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:05.524362 26928 hybrid_clock.cc:648] HybridClock initialized: now 1790303105524344 us; error 32 us; skew 500 ppm
Sep 25 02:25:05 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:05.526943 26928 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:25:05.527360 26928 webserver.cc:460] Webserver started at http://127.11.125.2:44559/ using document root <none> and password file <none>
I20260925 02:25:05.527601 26928 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:05.527647 26928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:05.529030 26928 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.529695 26944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.529913 26928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:05.530006 26928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1b9cc7d135b943f2902efd44d933f55a"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "56527cfdcd07f7dd73be554d3f798fe7"
server_key_iv: "aa4e72227686a95899b2c4dfab7a5231"
server_key_version: "encryptionkey@0"
I20260925 02:25:05.530341 26928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:05.555670 26928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:05.558061 26928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:05.558178 26928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:05.558370 26928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:05.558706 26928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:05.558766 26928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.558802 26928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:05.558859 26928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.564714 26928 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:37609
I20260925 02:25:05.564769 27058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:37609 every 8 connection(s)
I20260925 02:25:05.565176 26928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:25:05.566300 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.565914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:53615)
0925 02:25:05.566069 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.566151 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20260925 02:25:05.566711 27059 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44513 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:44513: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
I20260925 02:25:05.567013 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.566907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:52997)
0925 02:25:05.566958 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.566991 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:25:05.567291 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.567200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:36969)
0925 02:25:05.567245 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.567270 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:25:05.567441 27059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:25:05.567492 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 26928
I20260925 02:25:05.567768 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:39883
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:44513
--webserver_port=41265
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:05.649785 27062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:05.649983 27062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:05.650007 27062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:05.651188 27062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:05.651239 27062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:05.651269 27062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:25:05.652813 27062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:39883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=41265
--tserver_master_addrs=127.11.125.62:44513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:05.653082 27062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:05.653324 27062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:05.658583 27068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.658577 27062 server_base.cc:1034] running on GCE node
W20260925 02:25:05.658558 27067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:05.658607 27070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:05.659000 27062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:05.659256 27062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:05.660427 27062 hybrid_clock.cc:648] HybridClock initialized: now 1790303105660412 us; error 27 us; skew 500 ppm
Sep 25 02:25:05 dist-test-slave-hj7s krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:05.670387 27062 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:25:05.670863 27062 webserver.cc:460] Webserver started at http://127.11.125.3:41265/ using document root <none> and password file <none>
I20260925 02:25:05.671099 27062 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:05.671171 27062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:05.672648 27062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:25:05.673348 27077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.673527 27062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.673623 27062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2"
format_stamp: "Formatted at 2026-09-25 02:25:03 on dist-test-slave-hj7s"
server_key: "7f709c1ec1d269272d0c92414924b068"
server_key_iv: "805a40aca3ff89f64845bc6c56e9feb8"
server_key_version: "encryptionkey@0"
I20260925 02:25:05.673967 27062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:05.706434 27062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:05.708379 27062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:05.708494 27062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:05.708695 27062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:05.709095 27062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:05.709131 27062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.709175 27062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:05.709205 27062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:05.715063 27062 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:39883
I20260925 02:25:05.715148 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:39883 every 8 connection(s)
I20260925 02:25:05.715497 27062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:25:05.716609 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.716194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:55073)
0925 02:25:05.716387 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.716464 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
W20260925 02:25:05.717057 27194 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:44513 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:44513: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
I20260925 02:25:05.717339 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.717246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:43323)
0925 02:25:05.717294 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.717321 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:25:05.717581 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:05.717531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:58995)
0925 02:25:05.717551 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:05.717569 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 02:25:05.717693 27194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:25:05.723596 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27062
I20260925 02:25:06.500813 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:06.500575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:48305)
0925 02:25:06.500704 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:06.500741 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260925 02:25:06.568007 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:06.567740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:41967)
0925 02:25:06.567912 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:06.567949 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260925 02:25:06.718259 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:06.718032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:37167)
0925 02:25:06.718170 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:06.718209 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260925 02:25:07.208535 27201 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:07.208669 27201 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:07.209064 27201 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
Sep 25 02:25:07 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.2@KRBTEST.COM
Sep 25 02:25:07 dist-test-slave-hj7s krb5kdc[25602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303105, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.3@KRBTEST.COM
I20260925 02:25:07.221906 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.209368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:60871 (local address 127.11.125.2:37609)
0925 02:25:07.209546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:07.209553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:07.209574 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:25:07.210163 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:07.210166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:07.210229 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:07.210410 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:07.211598 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.213069 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.214611 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.215258 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.215378 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:07.218308 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:07.218313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:07.218314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:07.218327 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:07.220071 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:07.220573 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:07.220576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:07.220576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:07.220661 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:07.220995 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:07.220996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:07.220996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:07.221258 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:07.221354 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:07.221661 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 02:25:07.221788 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":50,"threads_started":1}
I20260925 02:25:07.221906 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.209234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37609 (local address 127.11.125.1:60871)
0925 02:25:07.209410 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:07.209422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:07.209482 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:07.210500 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:07.210504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:07.211117 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:07.211442 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:07.211451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.213181 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.213182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:07.214515 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:07.214518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.215378 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.215378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:07.215518 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:07.216158 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:07.216167 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:07.218167 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:07.220157 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:07.220165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:07.220167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:07.220484 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:07.220743 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:07.220744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:07.220745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:07.220877 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:07.221346 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:07.221354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:07.221492 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 02:25:07.221606 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:25:07.221785 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":86,"thread_start_us":56,"threads_started":1}
I20260925 02:25:07.222149 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.209324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:49565 (local address 127.11.125.3:39883)
0925 02:25:07.209519 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:07.209533 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:07.209574 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:25:07.210163 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:07.210166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:07.210229 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:07.210410 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:07.211598 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.213059 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.214611 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.215258 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.215383 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:07.218631 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:07.218635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:07.218635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:07.218646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:07.220628 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:07.221006 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:07.221009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:07.221009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:07.221104 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:07.221317 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:07.221318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:07.221319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:07.221551 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:07.221644 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:07.221836 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 02:25:07.221984 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260925 02:25:07.222021 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.209204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39883 (local address 127.11.125.1:49565)
0925 02:25:07.209332 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:07.209344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:07.209405 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:07.210500 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:07.210504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:07.211117 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:07.211442 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:07.211451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.213209 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.213209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:07.214515 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:07.214518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:07.215370 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:07.215372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:07.215518 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:07.216153 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:07.216167 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:07.218512 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:07.220714 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:07.220715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:07.220716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:07.220890 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:07.221180 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:07.221181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:07.221182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:07.221250 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:07.221628 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:07.221629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:07.221677 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:25:07.221769 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:25:07.221879 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":57,"thread_start_us":62,"threads_started":1}
W20260925 02:25:07.223307 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:07.223441 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:07.223483 26808 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:07.223598 27201 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:07.501469 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.501297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:59759)
0925 02:25:07.501360 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:07.501397 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:25:07.568720 27206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.568475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:60849)
0925 02:25:07.568619 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:07.568668 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260925 02:25:07.718993 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:07.718777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:53631)
0925 02:25:07.718903 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:07.718942 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 02:25:08.502134 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:08.501916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:38383)
0925 02:25:08.502047 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:08.502085 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260925 02:25:08.569387 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:08.569181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:34915)
0925 02:25:08.569301 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:08.569340 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260925 02:25:08.719691 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:08.719477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:49003)
0925 02:25:08.719608 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:08.719647 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 02:25:08.921016 27211 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:08.921145 27211 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:08.921362 27211 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
W20260925 02:25:08.921840 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:08.921929 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:08.921981 26808 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:08.922073 27211 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:09.502827 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:09.502575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:48167)
0925 02:25:09.502739 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:09.502778 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260925 02:25:09.570055 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:09.569822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:37323)
0925 02:25:09.569954 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:09.569994 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 02:25:09.720377 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:09.720129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:43791)
0925 02:25:09.720259 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:09.720306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260925 02:25:10.503683 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:10.503429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:36823)
0925 02:25:10.503588 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:10.503633 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260925 02:25:10.570736 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:10.570521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:37191)
0925 02:25:10.570658 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:10.570692 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260925 02:25:10.721166 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:10.720899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:40357)
0925 02:25:10.721087 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:10.721121 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260925 02:25:10.956079 27219 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:10.956197 27219 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:10.956449 27219 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
W20260925 02:25:10.956842 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:10.956984 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:10.957077 26808 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:10.957165 27219 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:11.504413 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:11.504149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:36265)
0925 02:25:11.504288 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:11.504333 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260925 02:25:11.571439 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:11.571176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:46403)
0925 02:25:11.571326 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:11.571388 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 02:25:11.721807 27222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:11.721605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:40491)
0925 02:25:11.721724 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:11.721764 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 02:25:12.505260 27223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:12.505026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:54543)
0925 02:25:12.505173 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:12.505210 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260925 02:25:12.572144 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:12.571915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:36297)
0925 02:25:12.572054 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:12.572095 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 02:25:12.722555 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:12.722308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:53003)
0925 02:25:12.722467 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:12.722509 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260925 02:25:12.831921 27226 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:12.832018 27226 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:12.832233 27226 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
W20260925 02:25:12.832849 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:12.832952 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:12.832995 26810 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:12.833118 27226 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:13.506111 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:13.505796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:41967)
0925 02:25:13.506003 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:13.506045 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260925 02:25:13.572885 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:13.572649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:47481)
0925 02:25:13.572789 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:13.572834 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260925 02:25:13.723172 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:13.722983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:35795)
0925 02:25:13.723096 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:13.723131 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 02:25:14.506896 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:14.506667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:37769)
0925 02:25:14.506795 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:14.506833 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260925 02:25:14.573607 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:14.573391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:47789)
0925 02:25:14.573525 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:14.573563 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260925 02:25:14.723802 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:14.723586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:42287)
0925 02:25:14.723723 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:14.723756 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260925 02:25:15.507777 27233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:15.507448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:42063)
0925 02:25:15.507676 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:15.507722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":122,"threads_started":1}
I20260925 02:25:15.574337 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:15.574088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:55987)
0925 02:25:15.574243 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:15.574288 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260925 02:25:15.724507 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:15.724273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:41189)
0925 02:25:15.724425 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:15.724463 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260925 02:25:15.919126 27236 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:15.919245 27236 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:15.919471 27236 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
W20260925 02:25:15.919834 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:15.919935 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:15.920024 26810 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:15.920101 27236 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:16.508490 27237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:16.508244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:55901)
0925 02:25:16.508398 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:16.508441 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260925 02:25:16.575126 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:16.574861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:49141)
0925 02:25:16.575001 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:16.575070 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260925 02:25:16.725133 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:16.724938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:54345)
0925 02:25:16.725056 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:16.725090 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 02:25:17.509384 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:17.509165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:55017)
0925 02:25:17.509289 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:17.509328 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260925 02:25:17.575812 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:17.575583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:59821)
0925 02:25:17.575725 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:17.575765 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260925 02:25:17.725823 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:17.725603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:43079)
0925 02:25:17.725727 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:17.725774 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 02:25:18.510116 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:18.509880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:56049)
0925 02:25:18.510026 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:18.510068 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260925 02:25:18.576637 27244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:18.576305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:47785)
0925 02:25:18.576544 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:18.576586 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260925 02:25:18.726549 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:18.726318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:46849)
0925 02:25:18.726458 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:18.726500 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260925 02:25:19.205070 27246 raft_consensus.cc:482] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:19.205215 27246 raft_consensus.cc:504] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fddafc6e7ce4eb385f3ca85fd642cd2" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39883 } } peers { permanent_uuid: "1b9cc7d135b943f2902efd44d933f55a" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37609 } } peers { permanent_uuid: "5a965e6137984fa188aa5c83218a5161" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 44775 } }
I20260925 02:25:19.205415 27246 leader_election.cc:288] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883), 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609)
W20260925 02:25:19.205997 26810 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9cc7d135b943f2902efd44d933f55a (127.11.125.2:37609): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
W20260925 02:25:19.206080 26808 leader_election.cc:341] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8fddafc6e7ce4eb385f3ca85fd642cd2 (127.11.125.3:39883): Not found: Tablet not found: 349e921cf8374f04bd98bc57021b7d8a
I20260925 02:25:19.206115 26808 leader_election.cc:302] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a965e6137984fa188aa5c83218a5161; no voters: 1b9cc7d135b943f2902efd44d933f55a, 8fddafc6e7ce4eb385f3ca85fd642cd2
I20260925 02:25:19.206210 27246 raft_consensus.cc:2726] T 349e921cf8374f04bd98bc57021b7d8a P 5a965e6137984fa188aa5c83218a5161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:25:19.510782 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:19.510584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:34735)
0925 02:25:19.510701 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:19.510734 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260925 02:25:19.577535 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:19.577304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:48575)
0925 02:25:19.577445 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:19.577487 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 02:25:19.727241 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:19.727011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:33123)
0925 02:25:19.727145 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:19.727195 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 02:25:20.511566 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:20.511346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.1:55605)
0925 02:25:20.511480 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:20.511518 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 02:25:20.578266 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:20.578039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.2:41237)
0925 02:25:20.578178 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:20.578216 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260925 02:25:20.728010 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:20.727762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44513 (local address 127.11.125.3:48125)
0925 02:25:20.727919 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:20.727950 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:44513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260925 02:25:20.733918 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26762
I20260925 02:25:20.739365 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 26928
I20260925 02:25:20.743853 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27062
2026-09-25T02:25:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18756 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:25:20 dist-test-slave-hj7s krb5kdc[27255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:25:20 dist-test-slave-hj7s krb5kdc[27255](info): set up 2 sockets
Sep 25 02:25:20 dist-test-slave-hj7s krb5kdc[27255](info): commencing operation
krb5kdc: starting...
W20260925 02:25:21.787251 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:25:21 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:25:21.904481 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:21.986800 27268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:21.986999 27268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:21.987021 27268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:21.988092 27268 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:25:21.988125 27268 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:25:21.988138 27268 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:25:21.988150 27268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:21.988162 27268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:21.988175 27268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:21.988188 27268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:21.988202 27268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:21.989686 27268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:21.989883 27268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:21.990049 27268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:25:21.995136 27268 server_base.cc:1034] running on GCE node
W20260925 02:25:21.995206 27274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:21.995142 27273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:21.995217 27276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:21.995599 27268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:21.995836 27268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:21.996971 27268 hybrid_clock.cc:648] HybridClock initialized: now 1790303121996961 us; error 33 us; skew 500 ppm
Sep 25 02:25:21 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303121, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:21.999359 27268 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:21.999814 27268 webserver.cc:460] Webserver started at http://127.11.125.62:42967/ using document root <none> and password file <none>
I20260925 02:25:22.000056 27268 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:22.000126 27268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:22.000281 27268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:22.001379 27268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "9634d2a175a241ce9bb370f6bafa387d"
format_stamp: "Formatted at 2026-09-25 02:25:21 on dist-test-slave-hj7s"
server_key: "851bd41ea67c4eed4268cf50273f9def"
server_key_iv: "0d1fd5b5fc4f34ea297f1a0cda3e797d"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.001741 27268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "9634d2a175a241ce9bb370f6bafa387d"
format_stamp: "Formatted at 2026-09-25 02:25:21 on dist-test-slave-hj7s"
server_key: "851bd41ea67c4eed4268cf50273f9def"
server_key_iv: "0d1fd5b5fc4f34ea297f1a0cda3e797d"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.003067 27268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:25:22.003783 27283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.003959 27268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:22.004067 27268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "9634d2a175a241ce9bb370f6bafa387d"
format_stamp: "Formatted at 2026-09-25 02:25:21 on dist-test-slave-hj7s"
server_key: "851bd41ea67c4eed4268cf50273f9def"
server_key_iv: "0d1fd5b5fc4f34ea297f1a0cda3e797d"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.004142 27268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:22.018975 27268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:22.020748 27268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:22.020958 27268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:22.024853 27268 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:43335
I20260925 02:25:22.024964 27337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:43335 every 8 connection(s)
I20260925 02:25:22.025346 27268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:22.026193 27338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:22.028546 27338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap starting.
I20260925 02:25:22.029150 27338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:22.029397 27338 log.cc:824] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:22.030049 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27268
I20260925 02:25:22.030164 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:25:22.030262 27338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: No bootstrap required, opened a new log
I20260925 02:25:22.030418 11764 external_mini_cluster.cc:1428] Setting key af31fe348c5664c76842e57a0d15b7c5
I20260925 02:25:22.032728 27338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:22.032866 27338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:22.032888 27338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Initialized, Role: FOLLOWER
I20260925 02:25:22.033070 27338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:22.033136 27338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:22.033159 27338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:22.033228 27338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:22.033733 27338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:22.033816 27338 leader_election.cc:302] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9634d2a175a241ce9bb370f6bafa387d; no voters:
I20260925 02:25:22.034044 27338 leader_election.cc:288] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:25:22.034086 27343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:22.034250 27343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 LEADER]: Becoming Leader. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Running, Role: LEADER
I20260925 02:25:22.034382 27338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:22.034399 27343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:22.034834 27345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9634d2a175a241ce9bb370f6bafa387d. Latest consensus state: current_term: 1 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:25:22.034927 27345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:22.035171 27344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:25:22.035485 27344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:22.035645 27350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:22.036159 27350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:22.038841 27350 catalog_manager.cc:1347] Generated new cluster ID: 97007d4d17d74d80b39dcf2c3c56e0cf
I20260925 02:25:22.038916 27350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:25:22.042579 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.030937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.11.125.62:43335)
0925 02:25:22.031134 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.031142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.031170 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:25:22.031669 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.031676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.031722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.031830 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:22.032629 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.034028 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.035084 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.035713 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.035816 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.038850 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:22.038854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:22.038855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:22.038866 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:22.040862 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.041291 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.041294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.041295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.041410 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.041611 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.041612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.041612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.041918 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:22.042078 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.042151 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.042300 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260925 02:25:22.056372 27350 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:25:22.057148 27350 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:25:22.069394 27350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Generated new TSK 0
I20260925 02:25:22.069612 27350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:25:22.102152 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:22.184564 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:22.184803 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:22.184841 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:22.186053 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:22.186120 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:22.186165 27364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:25:22.187651 27364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:22.187922 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:22.188148 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:22.193236 27372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:22.193267 27370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:22.193236 27369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:22.193264 27364 server_base.cc:1034] running on GCE node
I20260925 02:25:22.193670 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:22.193936 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:22.195101 27364 hybrid_clock.cc:648] HybridClock initialized: now 1790303122195086 us; error 28 us; skew 500 ppm
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:22.197597 27364 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:25:22.198024 27364 webserver.cc:460] Webserver started at http://127.11.125.1:41209/ using document root <none> and password file <none>
I20260925 02:25:22.198247 27364 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:22.198313 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:22.198470 27364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:22.199505 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "32c89f3d785841758a0bbb3ad440bcf1"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "98a2fa63f51dc0981224993f3001e47f"
server_key_iv: "3ce9a77e45719ee4fb5b77f0f175ba6a"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.199870 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "32c89f3d785841758a0bbb3ad440bcf1"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "98a2fa63f51dc0981224993f3001e47f"
server_key_iv: "3ce9a77e45719ee4fb5b77f0f175ba6a"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.201234 27364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:25:22.201952 27379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.202131 27364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:25:22.202239 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "32c89f3d785841758a0bbb3ad440bcf1"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "98a2fa63f51dc0981224993f3001e47f"
server_key_iv: "3ce9a77e45719ee4fb5b77f0f175ba6a"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.202314 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:22.212159 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:22.214133 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:22.214287 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:22.214533 27364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:22.214852 27364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:22.214907 27364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.214954 27364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:22.214978 27364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.221074 27364 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37741
I20260925 02:25:22.221176 27494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37741 every 8 connection(s)
I20260925 02:25:22.221493 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:25:22.227327 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27364
I20260925 02:25:22.227499 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:25:22.227728 11764 external_mini_cluster.cc:1428] Setting key b288d049df37eab2380eb3151a2bce55
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:22.231828 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.222221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:44163)
0925 02:25:22.222409 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.222447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.223028 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.223291 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.223296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.223898 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:22.224090 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.224096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.224835 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.224836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.225721 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.225724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.226179 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.226180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.226310 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.226792 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:22.226800 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:22.228624 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:22.230282 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.230290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.230293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.230620 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.230826 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.230827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.230828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.230977 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.231285 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:22.231290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:22.231447 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.231537 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.231662 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260925 02:25:22.231837 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.222237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:44163 (local address 127.11.125.62:43335)
0925 02:25:22.222280 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.222282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.222535 (+ 253us) server_negotiation.cc:407] Connection header received
0925 02:25:22.223147 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.223149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.223162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.223226 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:22.224207 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.224735 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.225832 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.226061 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.226163 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.228715 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:22.228717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:22.228717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:22.228723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:22.230201 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.230701 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.230701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.230702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.230755 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.231061 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.231062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.231063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.231214 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:22.231285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.231587 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.231692 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:25:22.232381 27495 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:22.232498 27495 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:22.232723 27495 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:22.233390 27302 ts_manager.cc:194] Registered new tserver with Master: 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:22.234411 27302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:44163
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:25:22.255906 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:22.339239 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:22.339478 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:22.339520 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:22.340679 27500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:22.340741 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:22.340786 27500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:25:22.342262 27500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:22.342527 27500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:22.342760 27500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:22.347775 27506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:22.347790 27508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:22.348021 27500 server_base.cc:1034] running on GCE node
W20260925 02:25:22.347790 27505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:22.348341 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:22.348601 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:22.349761 27500 hybrid_clock.cc:648] HybridClock initialized: now 1790303122349755 us; error 35 us; skew 500 ppm
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:22.352221 27500 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:25:22.352699 27500 webserver.cc:460] Webserver started at http://127.11.125.2:45327/ using document root <none> and password file <none>
I20260925 02:25:22.352979 27500 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:22.353048 27500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:22.353205 27500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:22.354228 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "ad6f4954ea1043629b8b6733baf53868"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "42e9751f14ac974118c875c59a8150e9"
server_key_iv: "e57b6dae2b6d8facabf8eb431e1c6359"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.354609 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6f4954ea1043629b8b6733baf53868"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "42e9751f14ac974118c875c59a8150e9"
server_key_iv: "e57b6dae2b6d8facabf8eb431e1c6359"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.355962 27500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:25:22.356688 27515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.356849 27500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:25:22.356974 27500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "ad6f4954ea1043629b8b6733baf53868"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "42e9751f14ac974118c875c59a8150e9"
server_key_iv: "e57b6dae2b6d8facabf8eb431e1c6359"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.357052 27500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:22.370193 27500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:22.372051 27500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:22.372218 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:22.372475 27500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:22.372794 27500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:22.372849 27500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.372887 27500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:22.372936 27500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.378885 27500 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:46347
I20260925 02:25:22.378996 27629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:46347 every 8 connection(s)
I20260925 02:25:22.379287 27500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:25:22.381071 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27500
I20260925 02:25:22.381196 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:25:22.381423 11764 external_mini_cluster.cc:1428] Setting key 68c35f353e86bd6b32e25fefb0ab7ac3
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:22.389801 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.380035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:43529 (local address 127.11.125.62:43335)
0925 02:25:22.380097 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.380099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.380314 (+ 215us) server_negotiation.cc:407] Connection header received
0925 02:25:22.381415 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.381416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.381430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.381510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:22.382267 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.383026 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.384069 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.384402 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.384484 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.386897 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:22.386898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:22.386899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:22.386906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:22.388324 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.388757 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.388758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.388758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.388822 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.389122 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.389123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.389124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.389249 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:22.389331 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.389622 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.389711 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:25:22.389985 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.379996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:43529)
0925 02:25:22.380180 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.380223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.381298 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.381606 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.381611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.381917 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:22.382171 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.382178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.383119 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.383120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.383978 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.383981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.384495 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.384496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.384596 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.385171 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:22.385181 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:22.386779 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:22.388438 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.388446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.388448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.388676 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.388897 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.388898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.388898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.389043 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.389321 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:22.389329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:22.389478 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.389581 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.389739 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:25:22.390704 27630 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:22.390816 27630 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:22.391002 27630 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:22.391561 27302 ts_manager.cc:194] Registered new tserver with Master: ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
I20260925 02:25:22.391952 27302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:43529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:25:22.411304 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:22.494329 27635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:22.494582 27635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:22.494621 27635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:22.495764 27635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:22.495834 27635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:22.495890 27635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:25:22.497388 27635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:22.497666 27635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:22.497907 27635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:22.502915 27640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:22.502909 27641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:22.503098 27635 server_base.cc:1034] running on GCE node
W20260925 02:25:22.502961 27643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:22.503427 27635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:22.503638 27635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:22.504772 27635 hybrid_clock.cc:648] HybridClock initialized: now 1790303122504753 us; error 35 us; skew 500 ppm
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:22.507439 27635 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:25:22.507920 27635 webserver.cc:460] Webserver started at http://127.11.125.3:45401/ using document root <none> and password file <none>
I20260925 02:25:22.508167 27635 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:22.508216 27635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:22.508414 27635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:25:22.509629 27635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "c876f1372ab848ce9394d0bcde654288"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "0fbfbda1726b44c027ef6174b5ec649e"
server_key_iv: "3f7351d07817094bc18594c4ae554bc3"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.510046 27635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "c876f1372ab848ce9394d0bcde654288"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "0fbfbda1726b44c027ef6174b5ec649e"
server_key_iv: "3f7351d07817094bc18594c4ae554bc3"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.511437 27635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:25:22.512163 27650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.512320 27635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:25:22.512405 27635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "c876f1372ab848ce9394d0bcde654288"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "0fbfbda1726b44c027ef6174b5ec649e"
server_key_iv: "3f7351d07817094bc18594c4ae554bc3"
server_key_version: "encryptionkey@0"
I20260925 02:25:22.512459 27635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:22.538470 27635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:22.540799 27635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:22.540970 27635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:22.541181 27635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:22.541515 27635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:22.541551 27635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.541603 27635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:22.541649 27635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:22.547523 27635 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:45615
I20260925 02:25:22.547600 27766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:45615 every 8 connection(s)
I20260925 02:25:22.548024 27635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:25:22 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303122, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:22.556423 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27635
I20260925 02:25:22.556519 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:25:22.556715 11764 external_mini_cluster.cc:1428] Setting key 2595978b58416eea0dc54b5e9fc64eb4
I20260925 02:25:22.557866 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.549262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34671 (local address 127.11.125.62:43335)
0925 02:25:22.549324 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.549326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.549556 (+ 230us) server_negotiation.cc:407] Connection header received
0925 02:25:22.550099 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.550101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.550115 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.550180 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:22.551147 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.551677 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.552681 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.552885 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.553001 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.555095 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:22.555097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:22.555098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:22.555105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:22.556250 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.556674 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.556676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.556676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.556746 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.557073 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.557074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.557075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.557240 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:22.557310 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.557698 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.557791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:25:22.558039 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.549253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:34671)
0925 02:25:22.549431 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.549470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.549989 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.550255 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.550259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.550839 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:22.551045 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.551050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.551782 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.551783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.552587 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.552589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.553015 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.553016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.553147 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.553612 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:22.553620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:22.554995 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:22.556334 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.556342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.556344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.556612 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.556814 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:22.556815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:22.556815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:22.556994 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:22.557322 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:22.557329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:22.557533 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.557654 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.557842 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260925 02:25:22.558490 27767 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:22.558602 27767 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:22.558784 27767 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:22.559171 27302 ts_manager.cc:194] Registered new tserver with Master: c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615)
I20260925 02:25:22.559551 27302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:34671
I20260925 02:25:22.568608 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:25:22.575866 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.570094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.11.125.62:43335)
0925 02:25:22.570169 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.570171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.570263 (+ 92us) server_negotiation.cc:407] Connection header received
0925 02:25:22.570302 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.570302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.570312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.570358 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:22.570876 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.571354 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.572084 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.572265 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.572315 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.573519 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:22.573520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:22.573521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:22.573527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:22.574945 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.575229 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.575230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.575230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.575282 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:22.575419 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:22.575420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:22.575420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:22.575556 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:22.575626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.575707 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.575767 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:25:22.577426 27302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:25:22.579036 27302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:25:22.585561 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.11.125.1:37741)
0925 02:25:22.582088 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.582093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.582101 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:25:22.582137 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.582139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.582155 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.582380 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.582453 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583456 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.584565 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585146 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585228 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585288 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.585440 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.585485 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":89,"threads_started":1}
I20260925 02:25:22.585552 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.0.0.1:39654)
0925 02:25:22.581928 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.581937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.581972 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.582248 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.582249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.582251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.582292 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.582294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583554 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.583555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.584483 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.584486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585228 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.585306 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585327 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.585419 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.585474 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":47,"thread_start_us":92,"threads_started":1}
I20260925 02:25:22.585767 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.0.0.1:53992)
0925 02:25:22.581608 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.581652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.581720 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.581969 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.581972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.581974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.582044 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.582047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583356 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.583357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.584490 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.584492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585525 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.585611 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585614 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.585659 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.585700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":3,"thread_start_us":44,"threads_started":1}
I20260925 02:25:22.586014 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.11.125.2:46347)
0925 02:25:22.581767 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.581773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.581786 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:25:22.581836 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.581839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.581866 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.581955 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.582119 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583250 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.584594 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585352 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585484 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.585881 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.585937 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 02:25:22.586061 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.0.0.1:51132)
0925 02:25:22.581765 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.581776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.581832 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.582179 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.582180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.582182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.582232 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.582234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583126 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.583127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.584494 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.584496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585524 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.585599 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585603 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.585949 (+ 346us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.585991 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260925 02:25:22.586366 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.581604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.11.125.3:45615)
0925 02:25:22.581806 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.581811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.581821 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:25:22.581901 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.581902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.581916 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.582010 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.582380 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.583023 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.584594 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.585352 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.585426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.585485 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.586236 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.586285 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260925 02:25:22.586431 27428 tablet_service.cc:1467] Processing CreateTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 (DEFAULT_TABLE table=test-table [id=7a12ef8d9d784a7ab1b2a45a2f018a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:22.586879 27428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a0bd1b15f4417fa3f4b0c30d953714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:22.586862 27563 tablet_service.cc:1467] Processing CreateTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 (DEFAULT_TABLE table=test-table [id=7a12ef8d9d784a7ab1b2a45a2f018a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:22.587200 27563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a0bd1b15f4417fa3f4b0c30d953714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:22.587182 27698 tablet_service.cc:1467] Processing CreateTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 (DEFAULT_TABLE table=test-table [id=7a12ef8d9d784a7ab1b2a45a2f018a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:22.587497 27698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a0bd1b15f4417fa3f4b0c30d953714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:22.589741 27784 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap starting.
I20260925 02:25:22.590499 27784 tablet_bootstrap.cc:654] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:22.590583 27785 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Bootstrap starting.
I20260925 02:25:22.590847 27786 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Bootstrap starting.
I20260925 02:25:22.591184 27784 log.cc:824] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:22.591329 27785 tablet_bootstrap.cc:654] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:22.591524 27786 tablet_bootstrap.cc:654] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:22.592017 27786 log.cc:824] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:22.592067 27784 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: No bootstrap required, opened a new log
I20260925 02:25:22.592089 27785 log.cc:824] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:22.592126 27784 ts_tablet_manager.cc:1397] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:22.592818 27786 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: No bootstrap required, opened a new log
I20260925 02:25:22.592940 27786 ts_tablet_manager.cc:1397] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:22.592958 27785 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: No bootstrap required, opened a new log
I20260925 02:25:22.593029 27785 ts_tablet_manager.cc:1397] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:25:22.594254 27784 raft_consensus.cc:348] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.594446 27784 raft_consensus.cc:374] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:22.594484 27784 raft_consensus.cc:729] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Initialized, Role: FOLLOWER
I20260925 02:25:22.594403 27785 raft_consensus.cc:348] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.594530 27785 raft_consensus.cc:374] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:22.594551 27785 raft_consensus.cc:729] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6f4954ea1043629b8b6733baf53868, State: Initialized, Role: FOLLOWER
I20260925 02:25:22.594605 27784 consensus_queue.cc:260] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.594712 27785 consensus_queue.cc:260] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.594890 27784 ts_tablet_manager.cc:1428] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:25:22.594928 27495 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
I20260925 02:25:22.594990 27785 ts_tablet_manager.cc:1428] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:22.594921 27786 raft_consensus.cc:348] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.595064 27786 raft_consensus.cc:374] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:22.595088 27786 raft_consensus.cc:729] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c876f1372ab848ce9394d0bcde654288, State: Initialized, Role: FOLLOWER
I20260925 02:25:22.595183 27786 consensus_queue.cc:260] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.595207 27630 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
I20260925 02:25:22.595428 27786 ts_tablet_manager.cc:1428] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:25:22.595438 27767 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
W20260925 02:25:22.629889 27631 tablet.cc:2369] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:25:22.655320 27790 raft_consensus.cc:482] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:22.655526 27790 raft_consensus.cc:504] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.656054 27790 leader_election.cc:288] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
I20260925 02:25:22.657073 27791 raft_consensus.cc:482] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:22.657193 27791 raft_consensus.cc:504] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.657533 27791 leader_election.cc:288] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:22.660254 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.656402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:37383 (local address 127.11.125.2:46347)
0925 02:25:22.656451 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.656453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.656556 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:25:22.656606 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.656607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.656617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.656691 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.656908 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.657537 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659191 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659818 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659883 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.659892 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.660148 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.660185 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":26}
I20260925 02:25:22.660344 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.656240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.11.125.1:36649)
0925 02:25:22.656287 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.656300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.656352 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.656568 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.656569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.656571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.656652 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.656654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.657665 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.657666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.659213 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.659215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.660037 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.660039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.660152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.660155 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.660206 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.660264 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":172}
I20260925 02:25:22.660610 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.656345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:36649 (local address 127.11.125.3:45615)
0925 02:25:22.656409 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.656412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.656420 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:25:22.656452 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.656453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.656463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.656530 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.656735 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.657550 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659314 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659919 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659993 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.660003 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.660500 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.660544 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:25:22.660878 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.656259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.11.125.1:37383)
0925 02:25:22.656479 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.656488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.656544 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.656726 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.656726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.656728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.656810 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.656811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.657653 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.657654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.659093 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.659096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659964 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.660042 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.660046 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.660644 (+ 598us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.660795 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":68,"thread_start_us":143,"threads_started":1}
I20260925 02:25:22.660925 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288" is_pre_election: true
I20260925 02:25:22.661103 27719 raft_consensus.cc:2454] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c89f3d785841758a0bbb3ad440bcf1 in term 0.
I20260925 02:25:22.661260 27583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6f4954ea1043629b8b6733baf53868" is_pre_election: true
I20260925 02:25:22.661340 27380 leader_election.cc:302] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c89f3d785841758a0bbb3ad440bcf1, c876f1372ab848ce9394d0bcde654288; no voters:
I20260925 02:25:22.661392 27583 raft_consensus.cc:2454] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c89f3d785841758a0bbb3ad440bcf1 in term 0.
I20260925 02:25:22.661449 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.657985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:55453 (local address 127.11.125.1:37741)
0925 02:25:22.658023 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.658025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.658138 (+ 113us) server_negotiation.cc:407] Connection header received
0925 02:25:22.658225 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.658225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.658233 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.658260 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.658445 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659099 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.660513 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.661092 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.661134 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.661141 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.661357 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.661392 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260925 02:25:22.661500 27790 raft_consensus.cc:2781] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:25:22.661550 27790 raft_consensus.cc:482] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:25:22.661580 27790 raft_consensus.cc:3037] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:22.661670 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.657750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.11.125.2:55453)
0925 02:25:22.658088 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.658097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.658159 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.658311 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.658311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.658313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.658379 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.658380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659191 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.660414 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.660417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.661199 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.661200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.661270 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.661273 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.661546 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.661601 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":5,"thread_start_us":105,"threads_started":1}
I20260925 02:25:22.661936 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.657657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.11.125.2:51133)
0925 02:25:22.657868 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.657878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:22.657917 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:22.658205 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:22.658206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:22.658208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:22.658266 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.658267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659359 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.659360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.660611 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:22.660613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.661689 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.661691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:22.661779 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.661783 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:22.661819 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 02:25:22.661864 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 02:25:22.661984 27449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c89f3d785841758a0bbb3ad440bcf1" is_pre_election: true
I20260925 02:25:22.662194 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.657882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:51133 (local address 127.11.125.3:45615)
0925 02:25:22.658048 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.658050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.658059 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:25:22.658082 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.658083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.658092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.658350 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:22.658387 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.659262 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.660754 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.661554 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.662050 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.662061 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.662084 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.662120 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260925 02:25:22.662374 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288" is_pre_election: true
I20260925 02:25:22.662456 27719 raft_consensus.cc:2454] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6f4954ea1043629b8b6733baf53868 in term 0.
I20260925 02:25:22.662613 27516 leader_election.cc:302] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad6f4954ea1043629b8b6733baf53868, c876f1372ab848ce9394d0bcde654288; no voters:
I20260925 02:25:22.662678 27790 raft_consensus.cc:504] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.662782 27791 raft_consensus.cc:2781] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:25:22.662863 27791 raft_consensus.cc:482] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:25:22.662895 27791 raft_consensus.cc:3037] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:22.662873 27790 leader_election.cc:288] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 1 election: Requested vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
I20260925 02:25:22.662884 27449 raft_consensus.cc:2379] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad6f4954ea1043629b8b6733baf53868 in current term 1: Already voted for candidate 32c89f3d785841758a0bbb3ad440bcf1 in this term.
I20260925 02:25:22.663038 27583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6f4954ea1043629b8b6733baf53868"
I20260925 02:25:22.663308 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288"
I20260925 02:25:22.663391 27719 raft_consensus.cc:3037] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:22.663977 27791 raft_consensus.cc:504] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.664155 27791 leader_election.cc:288] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 election: Requested vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:22.664160 27583 raft_consensus.cc:2379] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32c89f3d785841758a0bbb3ad440bcf1 in current term 1: Already voted for candidate ad6f4954ea1043629b8b6733baf53868 in this term.
I20260925 02:25:22.664307 27718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288"
I20260925 02:25:22.664420 27719 raft_consensus.cc:2454] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c89f3d785841758a0bbb3ad440bcf1 in term 1.
I20260925 02:25:22.664529 27449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a0bd1b15f4417fa3f4b0c30d953714" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c89f3d785841758a0bbb3ad440bcf1"
I20260925 02:25:22.664587 27380 leader_election.cc:302] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c89f3d785841758a0bbb3ad440bcf1, c876f1372ab848ce9394d0bcde654288; no voters:
I20260925 02:25:22.664600 27449 raft_consensus.cc:2379] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ad6f4954ea1043629b8b6733baf53868 in current term 1: Already voted for candidate 32c89f3d785841758a0bbb3ad440bcf1 in this term.
I20260925 02:25:22.664685 27790 raft_consensus.cc:2781] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:22.664850 27790 raft_consensus.cc:686] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 LEADER]: Becoming Leader. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Running, Role: LEADER
I20260925 02:25:22.664846 27518 leader_election.cc:302] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad6f4954ea1043629b8b6733baf53868; no voters: 32c89f3d785841758a0bbb3ad440bcf1, c876f1372ab848ce9394d0bcde654288
I20260925 02:25:22.664991 27790 consensus_queue.cc:237] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:22.665081 27791 raft_consensus.cc:2726] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:25:22.665901 27300 catalog_manager.cc:5495] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "32c89f3d785841758a0bbb3ad440bcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } health_report { overall_health: UNKNOWN } } }
I20260925 02:25:22.677210 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.674685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.11.125.1:37741)
0925 02:25:22.674754 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:22.674756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:22.674766 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:25:22.674830 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:22.674831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:22.674855 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:22.674930 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:22.675077 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.675558 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.676359 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:22.676546 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:22.676623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:22.676804 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:22.676949 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:25:22.677052 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:22.677071 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 02:25:22.677093 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:25:22.679051 27719 raft_consensus.cc:1261] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Refusing update from remote peer 32c89f3d785841758a0bbb3ad440bcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:22.679055 27583 raft_consensus.cc:1261] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Refusing update from remote peer 32c89f3d785841758a0bbb3ad440bcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:22.679324 27790 consensus_queue.cc:1035] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:22.679456 27796 consensus_queue.cc:1035] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:22.682121 27801 mvcc.cc:205] Tried to move back new op lower bound from 7333081590490755072 to 7333081590436466688. Current Snapshot: MvccSnapshot[applied={T|T < 7333081590490755072}]
I20260925 02:25:22.682354 27800 mvcc.cc:205] Tried to move back new op lower bound from 7333081590490755072 to 7333081590436466688. Current Snapshot: MvccSnapshot[applied={T|T < 7333081590490755072}]
I20260925 02:25:22.682533 27799 mvcc.cc:205] Tried to move back new op lower bound from 7333081590490755072 to 7333081590436466688. Current Snapshot: MvccSnapshot[applied={T|T < 7333081590490755072 or (T in {7333081590490755072})}]
I20260925 02:25:22.684520 27301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:22.684617 27301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:22.685464 27301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Sending DeleteTablet for 3 replicas of tablet 26a0bd1b15f4417fa3f4b0c30d953714
I20260925 02:25:22.685796 27428 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:39654
I20260925 02:25:22.685793 27698 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:51132
I20260925 02:25:22.685817 27563 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:53992
I20260925 02:25:22.686010 27810 tablet_replica.cc:330] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: stopping tablet replica
I20260925 02:25:22.686074 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27364
I20260925 02:25:22.686101 27810 raft_consensus.cc:2229] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:25:22.686271 27812 tablet_replica.cc:330] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: stopping tablet replica
I20260925 02:25:22.686457 27812 raft_consensus.cc:2229] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:22.686820 27812 raft_consensus.cc:2258] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:22.687235 27812 ts_tablet_manager.cc:1905] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:25:22.692510 27286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:37741 (error 108)
I20260925 02:25:22.692637 27812 ts_tablet_manager.cc:1918] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:22.692721 27812 log.cc:1196] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/26a0bd1b15f4417fa3f4b0c30d953714
I20260925 02:25:22.692713 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27500
I20260925 02:25:22.692829 27811 tablet_replica.cc:330] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: stopping tablet replica
I20260925 02:25:22.693038 27812 ts_tablet_manager.cc:1939] T 26a0bd1b15f4417fa3f4b0c30d953714 P ad6f4954ea1043629b8b6733baf53868: Deleting consensus metadata
I20260925 02:25:22.693576 27811 raft_consensus.cc:2229] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:22.693708 27811 raft_consensus.cc:2258] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:22.694358 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.694230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.0.0.1:39670)
0925 02:25:22.694276 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.694317 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 02:25:22.694479 27286 catalog_manager.cc:4584] TS 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26a0bd1b15f4417fa3f4b0c30d953714: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37741: connect: Connection refused (error 111)
I20260925 02:25:22.694772 27811 ts_tablet_manager.cc:1905] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:22.696166 27811 ts_tablet_manager.cc:1918] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:22.696228 27811 log.cc:1196] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/26a0bd1b15f4417fa3f4b0c30d953714
I20260925 02:25:22.696734 27811 ts_tablet_manager.cc:1939] T 26a0bd1b15f4417fa3f4b0c30d953714 P c876f1372ab848ce9394d0bcde654288: Deleting consensus metadata
I20260925 02:25:22.697207 27284 catalog_manager.cc:4841] TS c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615): tablet 26a0bd1b15f4417fa3f4b0c30d953714 (table test-table [id=7a12ef8d9d784a7ab1b2a45a2f018a69]) successfully deleted
I20260925 02:25:22.697957 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27635
I20260925 02:25:22.698026 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:22.697867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.0.0.1:54006)
0925 02:25:22.697945 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:22.697972 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.2:46347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:25:22.703510 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27268
I20260925 02:25:42.709069 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--webserver_interface=127.11.125.62
--webserver_port=42967
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:25:42.795295 27814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:42.795531 27814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:42.795571 27814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:42.796707 27814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:25:42.796772 27814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:25:42.796795 27814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:25:42.796834 27814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:25:42.796873 27814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:42.796936 27814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:42.796979 27814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:25:42.796999 27814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:25:42.798779 27814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=42967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:42.799041 27814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:42.799340 27814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:42.808535 27824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:42.808535 27821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:42.808597 27822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:42.808636 27814 server_base.cc:1034] running on GCE node
I20260925 02:25:42.809077 27814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:42.809350 27814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:42.810513 27814 hybrid_clock.cc:648] HybridClock initialized: now 1790303142810500 us; error 44 us; skew 500 ppm
Sep 25 02:25:42 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303142, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:42.813172 27814 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:25:42.813544 27814 webserver.cc:460] Webserver started at http://127.11.125.62:42967/ using document root <none> and password file <none>
I20260925 02:25:42.813730 27814 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:42.813778 27814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:42.815148 27814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:25:42.815764 27831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:42.815970 27814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:42.816061 27814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "9634d2a175a241ce9bb370f6bafa387d"
format_stamp: "Formatted at 2026-09-25 02:25:21 on dist-test-slave-hj7s"
server_key: "851bd41ea67c4eed4268cf50273f9def"
server_key_iv: "0d1fd5b5fc4f34ea297f1a0cda3e797d"
server_key_version: "encryptionkey@0"
I20260925 02:25:42.816426 27814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:42.848357 27814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:42.850852 27814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:42.850976 27814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:42.855345 27814 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:43335
I20260925 02:25:42.855414 27883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:43335 every 8 connection(s)
I20260925 02:25:42.855902 27814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:25:42.857597 27884 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:25:42.858273 27884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap starting.
I20260925 02:25:42.864897 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27814
I20260925 02:25:42.865280 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:37741
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=41209
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:25:42.868228 27884 log.cc:824] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:42.872004 27884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:42.872372 27884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap complete.
I20260925 02:25:42.874986 27884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:42.875259 27884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Initialized, Role: FOLLOWER
I20260925 02:25:42.875485 27884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:42.875592 27884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:25:42.875665 27884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:25:42.875731 27884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:25:42.876829 27884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:42.876986 27884 leader_election.cc:302] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9634d2a175a241ce9bb370f6bafa387d; no voters:
I20260925 02:25:42.877238 27884 leader_election.cc:288] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:25:42.877265 27888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:25:42.877400 27888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 LEADER]: Becoming Leader. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Running, Role: LEADER
I20260925 02:25:42.877537 27884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:25:42.877589 27888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:25:42.878026 27890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9634d2a175a241ce9bb370f6bafa387d. Latest consensus state: current_term: 2 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:25:42.878124 27890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:42.878043 27889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:25:42.878185 27889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:25:42.878393 27895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:25:42.879630 27895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:25:42.879827 27895 catalog_manager.cc:1259] Loaded cluster ID: 97007d4d17d74d80b39dcf2c3c56e0cf
I20260925 02:25:42.880060 27895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:25:42.881011 27895 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:25:42.881237 27895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Loaded TSK: 0
I20260925 02:25:42.892999 27895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Generated new TSK 1
I20260925 02:25:42.893127 27895 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:25:42.962227 27886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:42.962460 27886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:42.962497 27886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:42.963680 27886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:42.963750 27886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:42.963796 27886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:25:42.965368 27886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=41209
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:42.965633 27886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:42.965873 27886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:42.971892 27913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:42.971840 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:42.971928 27912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:42.972553 27886 server_base.cc:1034] running on GCE node
I20260925 02:25:42.972752 27886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:42.973057 27886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:42.974215 27886 hybrid_clock.cc:648] HybridClock initialized: now 1790303142974195 us; error 36 us; skew 500 ppm
Sep 25 02:25:42 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303142, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:42.977128 27886 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:25:42.977722 27886 webserver.cc:460] Webserver started at http://127.11.125.1:41209/ using document root <none> and password file <none>
I20260925 02:25:42.978005 27886 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:42.978155 27886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:42.979909 27886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:25:42.980739 27922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:42.980919 27886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:42.981024 27886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "32c89f3d785841758a0bbb3ad440bcf1"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "98a2fa63f51dc0981224993f3001e47f"
server_key_iv: "3ce9a77e45719ee4fb5b77f0f175ba6a"
server_key_version: "encryptionkey@0"
I20260925 02:25:42.981364 27886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:43.012789 27886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:43.015113 27886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:43.015285 27886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:43.015561 27886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:43.016086 27929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:25:43.017627 27886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:25:43.017678 27886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:25:43.017714 27886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:25:43.018450 27886 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:25:43.018497 27886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:43.018543 27929 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap starting.
I20260925 02:25:43.026971 27886 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37741
I20260925 02:25:43.027068 28038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37741 every 8 connection(s)
I20260925 02:25:43.027470 27886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:25:43.031023 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 27886
I20260925 02:25:43.031345 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:46347
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=45327
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:25:43.031733 27929 log.cc:824] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:43.034364 27929 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:25:43.034643 27929 tablet_bootstrap.cc:492] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap complete.
I20260925 02:25:43.035876 27929 ts_tablet_manager.cc:1397] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303142, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:43.036962 27929 raft_consensus.cc:348] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:43.037288 27929 raft_consensus.cc:729] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Initialized, Role: FOLLOWER
I20260925 02:25:43.037463 27929 consensus_queue.cc:260] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } }
I20260925 02:25:43.037752 27929 ts_tablet_manager.cc:1428] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:43.041569 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.028389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:57609)
0925 02:25:43.028795 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.028871 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.029702 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.030195 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.030201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.030617 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:43.030978 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.030985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.032752 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.032753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.034060 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.034063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.034734 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.034735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.034907 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.035520 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:43.035530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:43.037533 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:43.039909 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.039916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.039918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.040141 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.040489 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.040490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.040490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.040588 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.041106 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:43.041112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:43.041220 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.041297 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.041428 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
I20260925 02:25:43.041814 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.028592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:57609 (local address 127.11.125.62:43335)
0925 02:25:43.028835 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.028844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.029022 (+ 178us) server_negotiation.cc:407] Connection header received
0925 02:25:43.029940 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.029946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.029991 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.030106 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.031140 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.032567 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.034151 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.034625 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.034735 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.037665 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.037670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.037670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.037683 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.039818 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.040243 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.040247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.040248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.040412 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.040668 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.040669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.040670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.041034 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.041131 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.041409 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.041569 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260925 02:25:43.042452 28039 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:43.042567 28039 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:43.042766 28039 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:43.043685 27848 ts_manager.cc:194] Registered new tserver with Master: 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:43.045080 27848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:57609
I20260925 02:25:43.046240 28039 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303142, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:25:43.052623 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.044617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.11.125.1:37741)
0925 02:25:43.044787 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.044790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.044798 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:25:43.044882 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.044883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.044895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.045015 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.045465 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.046141 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.047213 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.047500 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.047579 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.049721 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.049730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.049730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.049738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.051260 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.051701 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.051704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.051705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.051765 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.051978 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.051979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.051979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.052127 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.052225 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.052429 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.052523 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260925 02:25:43.052623 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.044381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.0.0.1:45564)
0925 02:25:43.044555 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.044591 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.044658 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.045017 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.045024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.045297 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:43.045378 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.045381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.046235 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.046236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.047118 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.047120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.047579 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.047580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.047667 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.048130 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:43.048143 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:43.049628 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:43.051401 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.051407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.051409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.051612 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.051847 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.051848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.051848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.051914 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.052204 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:43.052212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:43.052266 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.052387 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.052523 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260925 02:25:43.053078 27972 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a0bd1b15f4417fa3f4b0c30d953714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:22 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:45564
I20260925 02:25:43.053603 28049 tablet_replica.cc:330] stopping tablet replica
I20260925 02:25:43.053718 28049 raft_consensus.cc:2229] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:43.053785 28049 raft_consensus.cc:2258] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:43.054102 28049 ts_tablet_manager.cc:1905] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:43.056710 28049 ts_tablet_manager.cc:1918] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:43.056787 28049 log.cc:1196] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal/wals/26a0bd1b15f4417fa3f4b0c30d953714
I20260925 02:25:43.057034 28049 ts_tablet_manager.cc:1939] T 26a0bd1b15f4417fa3f4b0c30d953714 P 32c89f3d785841758a0bbb3ad440bcf1: Deleting consensus metadata
I20260925 02:25:43.057556 27834 catalog_manager.cc:4841] TS 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741): tablet 26a0bd1b15f4417fa3f4b0c30d953714 (table test-table [id=7a12ef8d9d784a7ab1b2a45a2f018a69]) successfully deleted
W20260925 02:25:43.121568 28045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:43.121780 28045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:43.121806 28045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:43.123026 28045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:43.123071 28045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:43.123124 28045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:25:43.124624 28045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:46347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=45327
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:43.124879 28045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:43.125173 28045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:43.130347 28055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:43.130424 28054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:43.130481 28045 server_base.cc:1034] running on GCE node
W20260925 02:25:43.130347 28057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:43.130823 28045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:43.131136 28045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:43.132325 28045 hybrid_clock.cc:648] HybridClock initialized: now 1790303143132310 us; error 44 us; skew 500 ppm
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303143, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:43.134949 28045 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:25:43.135389 28045 webserver.cc:460] Webserver started at http://127.11.125.2:45327/ using document root <none> and password file <none>
I20260925 02:25:43.135622 28045 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:43.135706 28045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:43.137166 28045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:25:43.137806 28064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.137991 28045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:43.138092 28045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "ad6f4954ea1043629b8b6733baf53868"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "42e9751f14ac974118c875c59a8150e9"
server_key_iv: "e57b6dae2b6d8facabf8eb431e1c6359"
server_key_version: "encryptionkey@0"
I20260925 02:25:43.138468 28045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:43.186375 28045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:43.188506 28045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:43.188678 28045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:43.189026 28045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:43.189416 28045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:43.189476 28045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.189512 28045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:43.189560 28045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.195888 28045 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:46347
I20260925 02:25:43.195950 28177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:46347 every 8 connection(s)
I20260925 02:25:43.196338 28045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:25:43.198700 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28045
I20260925 02:25:43.199000 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:45615
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=45401
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303143, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:43.208729 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.197183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:50051 (local address 127.11.125.62:43335)
0925 02:25:43.197292 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.197295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.197511 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:25:43.198180 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.198181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.198196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.198274 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.199571 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.200229 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.201369 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.201586 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.201679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.204824 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.204826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.204826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.204843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.206810 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.207436 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.207438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.207439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.207518 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.207842 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.207843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.207844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.208006 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.208220 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.208525 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.208630 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:25:43.209035 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.197154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:50051)
0925 02:25:43.197367 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.197408 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.198036 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.198382 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.198386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.199189 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:43.199452 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.199458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.200355 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.200356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.201264 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.201267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.201693 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.201694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.201827 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.202533 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:43.202543 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:43.204686 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:43.206986 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.206997 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.207000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.207317 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.207611 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.207612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.207613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.207747 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.208207 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:43.208216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:43.208377 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.208500 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.208677 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260925 02:25:43.209594 28178 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:43.209744 28178 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:43.210026 28178 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:43.210565 27848 ts_manager.cc:194] Registered new tserver with Master: ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
I20260925 02:25:43.210968 27848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:50051
W20260925 02:25:43.289408 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:25:43.289613 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:25:43.289644 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:25:43.290812 28181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:25:43.290859 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:25:43.290892 28181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:25:43.292348 28181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=45401
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:25:43.292593 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:25:43.292837 28181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:25:43.297992 28188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:43.298084 28190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:25:43.298007 28187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:25:43.298246 28181 server_base.cc:1034] running on GCE node
I20260925 02:25:43.298513 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:25:43.298771 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:25:43.299955 28181 hybrid_clock.cc:648] HybridClock initialized: now 1790303143299935 us; error 39 us; skew 500 ppm
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303143, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:25:43.302580 28181 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:25:43.303073 28181 webserver.cc:460] Webserver started at http://127.11.125.3:45401/ using document root <none> and password file <none>
I20260925 02:25:43.303267 28181 fs_manager.cc:359] Metadata directory not provided
I20260925 02:25:43.303319 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:25:43.304658 28181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:25:43.305297 28197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.305485 28181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.305574 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "c876f1372ab848ce9394d0bcde654288"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "0fbfbda1726b44c027ef6174b5ec649e"
server_key_iv: "3f7351d07817094bc18594c4ae554bc3"
server_key_version: "encryptionkey@0"
I20260925 02:25:43.305909 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:25:43.320783 28181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:25:43.322852 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:25:43.322968 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:25:43.323175 28181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:25:43.323441 28181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:25:43.323472 28181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.323495 28181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:25:43.323508 28181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:25:43.330250 28181 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:45615
I20260925 02:25:43.330379 28312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:45615 every 8 connection(s)
I20260925 02:25:43.330721 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:25:43.334862 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28181
Sep 25 02:25:43 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303143, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:25:43.342412 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.331675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:41283 (local address 127.11.125.62:43335)
0925 02:25:43.331807 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.331809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.332020 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:25:43.332625 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.332627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.332643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.332700 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.333872 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.334424 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.336157 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.336381 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.336438 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.339103 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.339104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.339105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.339113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.340754 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.341348 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.341350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.341351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.341420 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.341685 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.341687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.341687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.341845 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.341947 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.342185 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.342304 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:25:43.342437 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.331671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:41283)
0925 02:25:43.331873 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.331925 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.332498 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.332779 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.332784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.333514 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:25:43.333745 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.333750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.334541 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.334542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.336071 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.336074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.336478 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.336479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.336684 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.337328 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:25:43.337338 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:25:43.339000 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:25:43.340859 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.340866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.340868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.341206 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.341491 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:25:43.341492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:25:43.341492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:25:43.341599 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:25:43.341917 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:25:43.341922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:25:43.342041 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.342142 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.342263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260925 02:25:43.343241 28313 heartbeater.cc:347] Connected to a master server at 127.11.125.62:43335
I20260925 02:25:43.343364 28313 heartbeater.cc:464] Registering TS with master...
I20260925 02:25:43.343561 28313 heartbeater.cc:511] Master 127.11.125.62:43335 requested a full tablet report, sending...
I20260925 02:25:43.343747 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.335428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.11.125.62:43335)
0925 02:25:43.335571 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.335574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.335722 (+ 148us) server_negotiation.cc:407] Connection header received
0925 02:25:43.335857 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.335858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.335870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.335933 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.336865 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.337424 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.338568 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.338872 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.338952 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.340194 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.340196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.340196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.340204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.342074 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.342881 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.342882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.342883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.342947 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.343176 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.343177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.343177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.343290 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.343486 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.343530 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.343626 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260925 02:25:43.344053 27847 ts_manager.cc:194] Registered new tserver with Master: c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615)
I20260925 02:25:43.344456 27847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:41283
I20260925 02:25:43.344625 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:25:43.347409 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.345107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.11.125.62:43335)
0925 02:25:43.345145 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.345146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.345313 (+ 167us) server_negotiation.cc:407] Connection header received
0925 02:25:43.345353 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.345354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.345392 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.345466 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:43.345566 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.346044 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.346728 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.346925 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.346978 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.347105 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:43.347145 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:25:43.347349 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38604: token expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:25:43.347476 28316 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38604: token expired
W20260925 02:25:43.347462 27770 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.125.62:43335)
I20260925 02:25:43.353053 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.347711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.11.125.62:43335)
0925 02:25:43.347751 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.347753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.347762 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:25:43.347823 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.347823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.347832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.347884 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:25:43.348301 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.348761 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.349507 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.349680 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.349742 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.350824 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:25:43.350824 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:25:43.350825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:25:43.350829 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:25:43.352014 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.352306 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.352306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.352307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.352353 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:25:43.352527 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:25:43.352527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:25:43.352528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:25:43.352614 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:25:43.352685 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.352871 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.352998 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:25:43.354249 27847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:25:43.355819 27847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:25:43.358114 27972 tablet_service.cc:1467] Processing CreateTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d (DEFAULT_TABLE table=test-table [id=321b13afb2194551b56f5c47a45c8e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:43.358328 27972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19c6cb49e084d70a47d2b317fa0a46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:43.359956 27929 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap starting.
I20260925 02:25:43.360277 27929 tablet_bootstrap.cc:654] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:43.360818 27929 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: No bootstrap required, opened a new log
I20260925 02:25:43.360873 27929 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:25:43.361248 27929 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.361332 27929 raft_consensus.cc:374] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:43.361390 27929 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Initialized, Role: FOLLOWER
I20260925 02:25:43.361436 27929 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.361536 27929 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:25:43.361987 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.358005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.11.125.3:45615)
0925 02:25:43.358217 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.358221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.358231 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:25:43.358272 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.358274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.358290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.358392 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:43.358487 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.359484 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.360793 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.361620 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.361704 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.361767 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.361895 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.361928 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260925 02:25:43.361990 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.357805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.0.0.1:44024)
0925 02:25:43.357872 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.357882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.357918 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.358393 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.358394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.358395 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:43.358440 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.358441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.359566 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.359567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.360714 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.360717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.361713 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.361714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.361787 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.361811 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.361880 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.361922 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:25:43.362742 28246 tablet_service.cc:1467] Processing CreateTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d (DEFAULT_TABLE table=test-table [id=321b13afb2194551b56f5c47a45c8e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:43.363094 28246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19c6cb49e084d70a47d2b317fa0a46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:43.363435 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.357875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.0.0.1:33162)
0925 02:25:43.358125 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.358137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.358192 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.359876 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.359878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.359879 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:43.359918 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.359919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.361249 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.361250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.362351 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.362354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.363255 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.363256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.363311 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.363314 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.363355 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.363387 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":72,"threads_started":1}
I20260925 02:25:43.363515 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.358110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.11.125.2:46347)
0925 02:25:43.359706 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.359711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.359721 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:25:43.359760 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.359762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.359779 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.359865 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:43.360025 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.361170 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.362432 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.363170 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.363250 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.363305 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.363381 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.363417 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":99,"threads_started":1}
I20260925 02:25:43.364219 28112 tablet_service.cc:1467] Processing CreateTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d (DEFAULT_TABLE table=test-table [id=321b13afb2194551b56f5c47a45c8e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:25:43.364580 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19c6cb49e084d70a47d2b317fa0a46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:25:43.366889 28323 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Bootstrap starting.
I20260925 02:25:43.367519 28323 tablet_bootstrap.cc:654] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:43.368077 28323 log.cc:824] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:43.368139 28325 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Bootstrap starting.
I20260925 02:25:43.368767 28325 tablet_bootstrap.cc:654] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Neither blocks nor log segments found. Creating new log.
I20260925 02:25:43.368957 28323 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: No bootstrap required, opened a new log
I20260925 02:25:43.369036 28323 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:25:43.369441 28325 log.cc:824] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Log is configured to *not* fsync() on all Append() calls
I20260925 02:25:43.370203 28325 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: No bootstrap required, opened a new log
I20260925 02:25:43.370277 28325 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:25:43.371277 28323 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.371492 28323 raft_consensus.cc:374] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:43.371539 28323 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c876f1372ab848ce9394d0bcde654288, State: Initialized, Role: FOLLOWER
I20260925 02:25:43.371691 28323 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.371984 28323 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:25:43.372008 28313 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
I20260925 02:25:43.372349 28325 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.372520 28325 raft_consensus.cc:374] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:25:43.372572 28325 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6f4954ea1043629b8b6733baf53868, State: Initialized, Role: FOLLOWER
I20260925 02:25:43.372699 28325 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.373024 28178 heartbeater.cc:503] Master 127.11.125.62:43335 was elected leader, sending a full tablet report...
I20260925 02:25:43.373034 28325 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:25:43.446374 28329 raft_consensus.cc:482] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:25:43.446519 28329 raft_consensus.cc:504] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
W20260925 02:25:43.446926 28179 tablet.cc:2369] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:25:43.447491 28329 leader_election.cc:288] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:43.450192 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.447326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:41053 (local address 127.11.125.3:45615)
0925 02:25:43.447383 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.447385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.447570 (+ 185us) server_negotiation.cc:407] Connection header received
0925 02:25:43.447704 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.447706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.447717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.447776 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:43.447960 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.448408 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449465 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.449823 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449885 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.449916 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.450063 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.450116 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:25:43.450213 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.447246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.11.125.2:41053)
0925 02:25:43.447505 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.447514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.447598 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.447806 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.447807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.447809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:43.447884 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.447886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.448528 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.448529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.449376 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.449379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.449916 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.449984 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.449988 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.450045 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.450114 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":79,"thread_start_us":131,"threads_started":1}
I20260925 02:25:43.450258 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.447377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:38515 (local address 127.11.125.1:37741)
0925 02:25:43.447422 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.447424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.447440 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:25:43.447470 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.447470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.447493 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.447579 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:25:43.447761 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.448380 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449368 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.449831 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449935 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.450021 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.450126 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.450177 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:25:43.450222 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.447241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.11.125.2:38515)
0925 02:25:43.447305 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.447320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:25:43.447375 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:25:43.447588 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:25:43.447589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:25:43.447591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:25:43.447675 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.447678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.448470 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.448471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.449271 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:25:43.449273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.449954 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.449955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:25:43.450041 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.450044 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:25:43.450104 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:25:43.450141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":29}
I20260925 02:25:43.450640 28266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288" is_pre_election: true
I20260925 02:25:43.450834 28266 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6f4954ea1043629b8b6733baf53868 in term 0.
I20260925 02:25:43.450748 27992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c89f3d785841758a0bbb3ad440bcf1" is_pre_election: true
I20260925 02:25:43.450970 27992 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6f4954ea1043629b8b6733baf53868 in term 0.
I20260925 02:25:43.451088 28065 leader_election.cc:302] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad6f4954ea1043629b8b6733baf53868, c876f1372ab848ce9394d0bcde654288; no voters:
I20260925 02:25:43.451231 28329 raft_consensus.cc:2781] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:25:43.451287 28329 raft_consensus.cc:482] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:25:43.451309 28329 raft_consensus.cc:3037] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:43.452031 28329 raft_consensus.cc:504] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.452181 28329 leader_election.cc:288] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 election: Requested vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:25:43.452375 28266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876f1372ab848ce9394d0bcde654288"
I20260925 02:25:43.452358 27992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "ad6f4954ea1043629b8b6733baf53868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c89f3d785841758a0bbb3ad440bcf1"
I20260925 02:25:43.452453 27992 raft_consensus.cc:3037] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:43.452454 28266 raft_consensus.cc:3037] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:25:43.453373 27992 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6f4954ea1043629b8b6733baf53868 in term 1.
I20260925 02:25:43.453374 28266 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6f4954ea1043629b8b6733baf53868 in term 1.
I20260925 02:25:43.453550 28065 leader_election.cc:302] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad6f4954ea1043629b8b6733baf53868, c876f1372ab848ce9394d0bcde654288; no voters:
I20260925 02:25:43.453661 28329 raft_consensus.cc:2781] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:25:43.453824 28329 raft_consensus.cc:686] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 LEADER]: Becoming Leader. State: Replica: ad6f4954ea1043629b8b6733baf53868, State: Running, Role: LEADER
I20260925 02:25:43.453948 28329 consensus_queue.cc:237] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:25:43.454770 27847 catalog_manager.cc:5495] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6f4954ea1043629b8b6733baf53868 (127.11.125.2). New cstate: current_term: 1 leader_uuid: "ad6f4954ea1043629b8b6733baf53868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } health_report { overall_health: UNKNOWN } } }
I20260925 02:25:43.479040 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.476288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.11.125.2:46347)
0925 02:25:43.476389 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:25:43.476391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:25:43.476399 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:25:43.476445 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:25:43.476446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:25:43.476465 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:25:43.476543 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:25:43.476681 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.477291 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.478102 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:25:43.478393 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:25:43.478472 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:25:43.478683 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:25:43.478794 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:25:43.478873 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:25:43.478935 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:25:43.478958 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:25:43.481032 27992 raft_consensus.cc:1261] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Refusing update from remote peer ad6f4954ea1043629b8b6733baf53868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:43.481045 28266 raft_consensus.cc:1261] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Refusing update from remote peer ad6f4954ea1043629b8b6733baf53868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:25:43.481392 28329 consensus_queue.cc:1035] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:43.481561 28331 consensus_queue.cc:1035] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:25:43.483884 28334 mvcc.cc:205] Tried to move back new op lower bound from 7333081675695108096 to 7333081675587960832. Current Snapshot: MvccSnapshot[applied={T|T < 7333081675695108096}]
I20260925 02:25:43.484467 28337 mvcc.cc:205] Tried to move back new op lower bound from 7333081675695108096 to 7333081675587960832. Current Snapshot: MvccSnapshot[applied={T|T < 7333081675695108096}]
I20260925 02:25:43.486809 27848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:43.486908 27848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:25:43.487691 27848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Sending DeleteTablet for 3 replicas of tablet d19c6cb49e084d70a47d2b317fa0a46d
I20260925 02:25:43.487995 28246 tablet_service.cc:1514] Processing DeleteTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:43 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:44024
I20260925 02:25:43.488123 27972 tablet_service.cc:1514] Processing DeleteTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:43 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:45564
I20260925 02:25:43.488162 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet d19c6cb49e084d70a47d2b317fa0a46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:25:43 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260925 02:25:43.488266 28049 tablet_replica.cc:330] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: stopping tablet replica
I20260925 02:25:43.488304 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27886
I20260925 02:25:43.488328 28049 raft_consensus.cc:2229] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:43.488389 28049 raft_consensus.cc:2258] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:43.489205 28344 tablet_replica.cc:330] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: stopping tablet replica
I20260925 02:25:43.489326 28344 raft_consensus.cc:2229] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:25:43.489449 28344 raft_consensus.cc:2258] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:43.489912 28344 ts_tablet_manager.cc:1905] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:25:43.492030 28345 tablet_replica.cc:330] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: stopping tablet replica
I20260925 02:25:43.492063 28344 ts_tablet_manager.cc:1918] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:25:43.492193 28345 raft_consensus.cc:2229] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:25:43.492228 28344 log.cc:1196] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/d19c6cb49e084d70a47d2b317fa0a46d
I20260925 02:25:43.492398 28345 raft_consensus.cc:2258] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:25:43.492640 28344 ts_tablet_manager.cc:1939] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Deleting consensus metadata
I20260925 02:25:43.493129 27832 catalog_manager.cc:4841] TS c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615): tablet d19c6cb49e084d70a47d2b317fa0a46d (table test-table [id=321b13afb2194551b56f5c47a45c8e56]) successfully deleted
W20260925 02:25:43.495404 27834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:37741 (error 108)
I20260925 02:25:43.495813 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28045
I20260925 02:25:43.497196 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.496606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.0.0.1:45568)
0925 02:25:43.497109 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.497160 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477}
W20260925 02:25:43.497866 27834 catalog_manager.cc:4584] TS 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d19c6cb49e084d70a47d2b317fa0a46d: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37741: connect: Connection refused (error 111)
I20260925 02:25:43.501209 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:25:43.501087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.0.0.1:33192)
0925 02:25:43.501139 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:25:43.501171 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.2:46347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:25:43.501276 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28181
I20260925 02:25:43.505950 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 27814
I20260925 02:26:03.513041 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--webserver_interface=127.11.125.62
--webserver_port=42967
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:03.595244 28347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:03.595486 28347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:03.595525 28347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:03.596691 28347 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:26:03.596755 28347 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:26:03.596776 28347 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:26:03.596807 28347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:03.596836 28347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:03.596885 28347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:03.596940 28347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:03.596987 28347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:03.598513 28347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=42967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:03.598806 28347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:03.599038 28347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:03.604454 28353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:03.604501 28355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:03.604467 28347 server_base.cc:1034] running on GCE node
W20260925 02:26:03.604507 28352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:03.605007 28347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:03.605281 28347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:03.606449 28347 hybrid_clock.cc:648] HybridClock initialized: now 1790303163606438 us; error 29 us; skew 500 ppm
Sep 25 02:26:03 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:03.609208 28347 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:03.609658 28347 webserver.cc:460] Webserver started at http://127.11.125.62:42967/ using document root <none> and password file <none>
I20260925 02:26:03.609905 28347 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:03.609952 28347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:03.611315 28347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:03.612041 28362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:03.612227 28347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:03.612322 28347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "9634d2a175a241ce9bb370f6bafa387d"
format_stamp: "Formatted at 2026-09-25 02:25:21 on dist-test-slave-hj7s"
server_key: "851bd41ea67c4eed4268cf50273f9def"
server_key_iv: "0d1fd5b5fc4f34ea297f1a0cda3e797d"
server_key_version: "encryptionkey@0"
I20260925 02:26:03.612692 28347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:03.625798 28347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:03.627678 28347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:03.627844 28347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:03.631843 28347 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:43335
I20260925 02:26:03.631901 28414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:43335 every 8 connection(s)
I20260925 02:26:03.632339 28347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:03.634003 28415 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:26:03.634682 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap starting.
I20260925 02:26:03.638172 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28347
I20260925 02:26:03.638477 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:37741
--local_ip_for_outbound_sockets=127.11.125.1
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=41209
--webserver_interface=127.11.125.1
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:03.644318 28415 log.cc:824] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:03.646643 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:26:03.646955 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Bootstrap complete.
I20260925 02:26:03.649289 28415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:26:03.649539 28415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Initialized, Role: FOLLOWER
I20260925 02:26:03.649724 28415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:26:03.649806 28415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:03.649883 28415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:03.649936 28415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:26:03.650885 28415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:26:03.651010 28415 leader_election.cc:302] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9634d2a175a241ce9bb370f6bafa387d; no voters:
I20260925 02:26:03.651216 28415 leader_election.cc:288] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:26:03.651250 28419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:26:03.651388 28419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [term 3 LEADER]: Becoming Leader. State: Replica: 9634d2a175a241ce9bb370f6bafa387d, State: Running, Role: LEADER
I20260925 02:26:03.651501 28415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:03.651544 28419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } }
I20260925 02:26:03.651928 28421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9634d2a175a241ce9bb370f6bafa387d. Latest consensus state: current_term: 3 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:26:03.651942 28420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9634d2a175a241ce9bb370f6bafa387d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9634d2a175a241ce9bb370f6bafa387d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43335 } } }
I20260925 02:26:03.652027 28421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:03.652056 28420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:03.652588 28427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:03.653390 28427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:03.653488 28427 catalog_manager.cc:1259] Loaded cluster ID: 97007d4d17d74d80b39dcf2c3c56e0cf
I20260925 02:26:03.653523 28427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:03.654796 28427 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:03.654963 28427 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Loaded TSK: 0
I20260925 02:26:03.654999 28427 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9634d2a175a241ce9bb370f6bafa387d: Loaded TSK: 1
F20260925 02:26:03.655246 28427 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efe301189ad google::LogMessage::Fail() at ??:0
@ 0x7efe3011b88a google::LogMessage::SendToLog() at ??:0
@ 0x7efe301184de google::LogMessage::Flush() at ??:0
@ 0x7efe30119a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efe32885104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efe32882d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efe380721cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efe3807248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efe38072594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7efe38038c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7efe38042ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efe31fe60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efe31fdb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efe32e3f6db start_thread at ??:0
@ 0x7efe2ec1661f clone at ??:0
W20260925 02:26:03.745823 28417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:03.746021 28417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:03.746047 28417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:03.747202 28417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:03.747242 28417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:03.747268 28417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:03.748700 28417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=41209
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:03.748950 28417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:03.749208 28417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:03.754534 28456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:03.754703 28417 server_base.cc:1034] running on GCE node
W20260925 02:26:03.754566 28457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:03.754536 28459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:03.755028 28417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:03.755383 28417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:03.756590 28417 hybrid_clock.cc:648] HybridClock initialized: now 1790303163756567 us; error 53 us; skew 500 ppm
Sep 25 02:26:03 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:03.759953 28417 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:26:03.760584 28417 webserver.cc:460] Webserver started at http://127.11.125.1:41209/ using document root <none> and password file <none>
I20260925 02:26:03.760870 28417 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:03.760983 28417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:03.762926 28417 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:03.763844 28466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:03.764079 28417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:03.764245 28417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "32c89f3d785841758a0bbb3ad440bcf1"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "98a2fa63f51dc0981224993f3001e47f"
server_key_iv: "3ce9a77e45719ee4fb5b77f0f175ba6a"
server_key_version: "encryptionkey@0"
I20260925 02:26:03.764720 28417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:03.804096 28417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:03.806728 28417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:03.806885 28417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:03.807179 28417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:03.807711 28473 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:26:03.809365 28417 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:26:03.809427 28417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:26:03.809463 28417 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:26:03.810334 28417 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:26:03.810381 28417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:26:03.810499 28473 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap starting.
I20260925 02:26:03.817740 28417 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37741
I20260925 02:26:03.817800 28583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37741 every 8 connection(s)
I20260925 02:26:03.818194 28417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:03.821470 28473 log.cc:824] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:03.823597 28473 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:26:03.823891 28473 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Bootstrap complete.
I20260925 02:26:03.824823 28473 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260925 02:26:03.825131 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28417
I20260925 02:26:03.825460 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:46347
--local_ip_for_outbound_sockets=127.11.125.2
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=45327
--webserver_interface=127.11.125.2
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:03.826092 28473 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:03.826345 28473 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Initialized, Role: FOLLOWER
I20260925 02:26:03.826574 28473 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:03.826867 28473 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
W20260925 02:26:03.885741 28586 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:03.818912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:37639)
0925 02:26:03.819151 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.819205 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:03.819940 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:03.885513 (+ 65573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
W20260925 02:26:03.885994 28584 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:43335 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:43335: Client connection negotiation failed: client connection to 127.11.125.62:43335: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:26:03.886348 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:03.886219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:54655)
0925 02:26:03.886280 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.886328 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 02:26:03.886724 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:03.886632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:36545)
0925 02:26:03.886680 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.886704 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 02:26:03.886871 28584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:26:03.915326 28589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:03.915565 28589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:03.915607 28589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:03.916772 28589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:03.916842 28589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:03.916887 28589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:03.918417 28589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:46347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=45327
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:03.918711 28589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:03.918962 28589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:03.924046 28597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:03.924064 28595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:03.924052 28594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:03.924324 28589 server_base.cc:1034] running on GCE node
I20260925 02:26:03.924482 28589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:03.924678 28589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:03.925879 28589 hybrid_clock.cc:648] HybridClock initialized: now 1790303163925840 us; error 28 us; skew 500 ppm
Sep 25 02:26:03 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:03.928376 28589 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:26:03.928853 28589 webserver.cc:460] Webserver started at http://127.11.125.2:45327/ using document root <none> and password file <none>
I20260925 02:26:03.929129 28589 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:03.929203 28589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:03.930583 28589 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:26:03.931279 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:03.931463 28589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:03.931556 28589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "ad6f4954ea1043629b8b6733baf53868"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "42e9751f14ac974118c875c59a8150e9"
server_key_iv: "e57b6dae2b6d8facabf8eb431e1c6359"
server_key_version: "encryptionkey@0"
I20260925 02:26:03.931942 28589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:03.948185 28589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:03.950301 28589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:03.950484 28589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:03.950749 28589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:03.951270 28612 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:26:03.952803 28589 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:26:03.952873 28589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 02:26:03.952962 28589 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:26:03.953621 28589 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:26:03.953679 28589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 02:26:03.953799 28612 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Bootstrap starting.
I20260925 02:26:03.955642 28612 log.cc:824] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:03.958267 28612 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:26:03.958679 28612 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Bootstrap complete.
I20260925 02:26:03.958870 28612 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:26:03.960299 28612 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:03.960577 28612 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6f4954ea1043629b8b6733baf53868, State: Initialized, Role: FOLLOWER
I20260925 02:26:03.960816 28612 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:03.961220 28612 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 02:26:03.961424 28589 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:46347
I20260925 02:26:03.961508 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:46347 every 8 connection(s)
I20260925 02:26:03.961803 28589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:26:03.962890 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:03.962519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:36131)
0925 02:26:03.962689 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.962767 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260925 02:26:03.963198 28724 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:43335 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:43335: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
I20260925 02:26:03.963519 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:03.963411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:42285)
0925 02:26:03.963463 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.963491 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:26:03.963894 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:03.963792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:57731)
0925 02:26:03.963841 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:03.963866 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:26:03.964007 28724 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:26:03.971328 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28589
I20260925 02:26:03.971662 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:45615
--local_ip_for_outbound_sockets=127.11.125.3
--tserver_master_addrs=127.11.125.62:43335
--webserver_port=45401
--webserver_interface=127.11.125.3
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:04.053606 28727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:04.053833 28727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:04.053882 28727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:04.055001 28727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:04.055064 28727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:04.055109 28727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:04.056538 28727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=45401
--tserver_master_addrs=127.11.125.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:04.056803 28727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:04.057047 28727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:04.061893 28735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:04.061933 28732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:04.062026 28727 server_base.cc:1034] running on GCE node
W20260925 02:26:04.061944 28733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:04.062397 28727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:04.062639 28727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:04.063796 28727 hybrid_clock.cc:648] HybridClock initialized: now 1790303164063792 us; error 35 us; skew 500 ppm
Sep 25 02:26:04 dist-test-slave-hj7s krb5kdc[27255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:04.066253 28727 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:26:04.066725 28727 webserver.cc:460] Webserver started at http://127.11.125.3:45401/ using document root <none> and password file <none>
I20260925 02:26:04.066958 28727 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:04.067029 28727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:04.068343 28727 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:04.069111 28742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:04.069340 28727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:04.069442 28727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "c876f1372ab848ce9394d0bcde654288"
format_stamp: "Formatted at 2026-09-25 02:25:22 on dist-test-slave-hj7s"
server_key: "0fbfbda1726b44c027ef6174b5ec649e"
server_key_iv: "3f7351d07817094bc18594c4ae554bc3"
server_key_version: "encryptionkey@0"
I20260925 02:26:04.069895 28727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:04.084860 28727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:04.086694 28727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:04.086865 28727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:04.087101 28727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:04.087466 28727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:04.087528 28727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:04.087570 28727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:04.087598 28727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:04.093477 28727 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:45615
I20260925 02:26:04.093531 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:45615 every 8 connection(s)
I20260925 02:26:04.093865 28727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:26:04.094839 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:04.094473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:47497)
0925 02:26:04.094623 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:04.094696 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260925 02:26:04.095263 28857 heartbeater.cc:644] Failed to heartbeat to 127.11.125.62:43335 (0 consecutive failures): Network error: Failed to ping master at 127.11.125.62:43335: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
I20260925 02:26:04.095530 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:04.095440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:46551)
0925 02:26:04.095484 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:04.095511 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 02:26:04.095798 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:04.095725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:35983)
0925 02:26:04.095763 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:04.095784 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 02:26:04.095922 28857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:26:04.096761 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28727
I20260925 02:26:04.887511 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:04.887257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:56807)
0925 02:26:04.887400 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:04.887441 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260925 02:26:04.964488 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:04.964300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:47839)
0925 02:26:04.964409 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:04.964442 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260925 02:26:05.061614 28863 raft_consensus.cc:482] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:05.061786 28863 raft_consensus.cc:504] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:05.062150 28863 leader_election.cc:288] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
Sep 25 02:26:05 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.3@KRBTEST.COM
Sep 25 02:26:05 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303163, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.2@KRBTEST.COM
I20260925 02:26:05.074972 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.062296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45615 (local address 127.11.125.1:33165)
0925 02:26:05.062341 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.062381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:05.062447 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:05.063407 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:05.063407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:05.063564 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:05.063659 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.063666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.065515 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.065517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.066865 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.066869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.067738 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.067739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.067935 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.068592 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:05.068604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:05.070881 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:05.073161 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073191 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.073193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.073605 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.073873 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.073875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.074013 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.074409 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:05.074418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:05.074552 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 02:26:05.074658 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:26:05.074867 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":106}
I20260925 02:26:05.074985 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.062449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33523 (local address 127.11.125.2:46347)
0925 02:26:05.062585 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:05.062600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:05.062720 (+ 120us) server_negotiation.cc:407] Connection header received
0925 02:26:05.062810 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:05.062812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:05.062877 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:05.063010 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:05.063784 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.065451 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.066964 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.067582 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.067698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.071277 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:05.071281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:05.071282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:05.071302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:05.073250 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073746 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.073749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.073749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.073890 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.074115 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.074116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.074116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.074349 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:05.074452 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:05.074721 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 02:26:05.074853 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260925 02:26:05.074987 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.062474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33165 (local address 127.11.125.3:45615)
0925 02:26:05.062609 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:05.062616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:05.062638 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:26:05.063234 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:05.063236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:05.063289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:05.063395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:05.063789 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.065354 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.066964 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.067582 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.067698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.071012 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:05.071016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:05.071017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:05.071030 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:05.073078 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073667 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.073671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.073671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.073785 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.074080 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.074081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.074081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.074330 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:05.074395 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:05.074726 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:26:05.074858 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260925 02:26:05.075105 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.062296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:46347 (local address 127.11.125.1:33523)
0925 02:26:05.062609 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.062628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:05.062683 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:05.063072 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:05.063074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:05.063390 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:05.063659 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.063666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.065562 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.065563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.066865 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.066869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.067736 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.067738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.067935 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.068595 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:05.068604 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:05.071172 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:05.073328 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.073329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.073650 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.073970 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.073971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.073971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.074027 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.074450 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:05.074451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:05.074551 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 02:26:05.074660 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:26:05.074953 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":312,"thread_start_us":66,"threads_started":1}
I20260925 02:26:05.076083 28676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ad6f4954ea1043629b8b6733baf53868" is_pre_election: true
I20260925 02:26:05.076251 28676 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c89f3d785841758a0bbb3ad440bcf1 in term 1.
W20260925 02:26:05.076377 28467 leader_election.cc:341] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615): Not found: Tablet not found: d19c6cb49e084d70a47d2b317fa0a46d
I20260925 02:26:05.076551 28470 leader_election.cc:302] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32c89f3d785841758a0bbb3ad440bcf1, ad6f4954ea1043629b8b6733baf53868; no voters: c876f1372ab848ce9394d0bcde654288
I20260925 02:26:05.076678 28863 raft_consensus.cc:2781] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:26:05.076761 28863 raft_consensus.cc:482] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:05.076787 28863 raft_consensus.cc:3037] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:26:05.078025 28863 raft_consensus.cc:504] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:05.078186 28863 leader_election.cc:288] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 election: Requested vote from peers c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615), ad6f4954ea1043629b8b6733baf53868 (127.11.125.2:46347)
I20260925 02:26:05.078382 28676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19c6cb49e084d70a47d2b317fa0a46d" candidate_uuid: "32c89f3d785841758a0bbb3ad440bcf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ad6f4954ea1043629b8b6733baf53868"
W20260925 02:26:05.078470 28467 leader_election.cc:341] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615): Not found: Tablet not found: d19c6cb49e084d70a47d2b317fa0a46d
I20260925 02:26:05.078471 28676 raft_consensus.cc:3037] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:26:05.079619 28676 raft_consensus.cc:2454] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c89f3d785841758a0bbb3ad440bcf1 in term 2.
I20260925 02:26:05.079778 28470 leader_election.cc:302] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32c89f3d785841758a0bbb3ad440bcf1, ad6f4954ea1043629b8b6733baf53868; no voters: c876f1372ab848ce9394d0bcde654288
I20260925 02:26:05.079902 28863 raft_consensus.cc:2781] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:26:05.080030 28863 raft_consensus.cc:686] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [term 2 LEADER]: Becoming Leader. State: Replica: 32c89f3d785841758a0bbb3ad440bcf1, State: Running, Role: LEADER
I20260925 02:26:05.080122 28863 consensus_queue.cc:237] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:05.080410 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.080275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:34995)
0925 02:26:05.080334 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.080381 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 02:26:05.096529 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.096259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:40899)
0925 02:26:05.096406 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.096457 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 02:26:05.456780 28676 raft_consensus.cc:1261] T d19c6cb49e084d70a47d2b317fa0a46d P ad6f4954ea1043629b8b6733baf53868 [term 2 FOLLOWER]: Refusing update from remote peer 32c89f3d785841758a0bbb3ad440bcf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 02:26:05.457237 28863 consensus_queue.cc:1035] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 02:26:05.458135 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.458018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:45479)
0925 02:26:05.458071 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.458106 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:26:05.459239 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.459151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:37503)
0925 02:26:05.459182 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.459210 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 02:26:05.459295 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.459182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:38649)
0925 02:26:05.459219 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.459248 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 02:26:05.459254 28467 consensus_peers.cc:487] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1 -> Peer c876f1372ab848ce9394d0bcde654288 (127.11.125.3:45615): Couldn't send request to peer c876f1372ab848ce9394d0bcde654288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d19c6cb49e084d70a47d2b317fa0a46d. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:26:05.863256 28875 ts_tablet_manager.cc:927] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Initiating tablet copy from peer 32c89f3d785841758a0bbb3ad440bcf1 (127.11.125.1:37741)
I20260925 02:26:05.863502 28875 tablet_copy_client.cc:316] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: tablet copy: Beginning tablet copy session from remote peer at address 127.11.125.1:37741
Sep 25 02:26:05 dist-test-slave-hj7s krb5kdc[27255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303164, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.1@KRBTEST.COM
I20260925 02:26:05.872221 28878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.863919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:34877 (local address 127.11.125.1:37741)
0925 02:26:05.864139 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:05.864144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:05.864157 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:26:05.864190 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:05.864192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:05.864209 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:05.864288 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:05.864832 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.865621 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.866525 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.866814 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.866883 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.869188 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:05.869199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:05.869199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:05.869208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:05.870792 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.871273 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.871277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.871278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.871347 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:05.871580 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:05.871581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:05.871582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:05.871739 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:05.871831 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:05.872031 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:26:05.872119 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260925 02:26:05.872231 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.863820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37741 (local address 127.11.125.3:34877)
0925 02:26:05.864043 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.864064 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:05.864132 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:05.864317 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:05.864325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:05.864691 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:05.864753 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.864757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.865737 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.865739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.866436 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:05.866439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:05.866897 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:05.866898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:05.866964 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:05.867454 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:05.867459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:05.869073 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:05.870935 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.870942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.870945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.871167 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.871432 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:05.871433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:05.871434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:05.871497 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:05.871819 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:05.871826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:05.871881 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 02:26:05.871993 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:26:05.872118 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":134,"threads_started":1}
I20260925 02:26:05.872848 28557 tablet_copy_service.cc:140] P 32c89f3d785841758a0bbb3ad440bcf1: Received BeginTabletCopySession request for tablet d19c6cb49e084d70a47d2b317fa0a46d from peer c876f1372ab848ce9394d0bcde654288 ({username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:34877)
I20260925 02:26:05.872954 28557 tablet_copy_service.cc:161] P 32c89f3d785841758a0bbb3ad440bcf1: Beginning new tablet copy session on tablet d19c6cb49e084d70a47d2b317fa0a46d from peer c876f1372ab848ce9394d0bcde654288 at {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:34877: session id = c876f1372ab848ce9394d0bcde654288-d19c6cb49e084d70a47d2b317fa0a46d
I20260925 02:26:05.873559 28557 tablet_copy_source_session.cc:205] T d19c6cb49e084d70a47d2b317fa0a46d P 32c89f3d785841758a0bbb3ad440bcf1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:26:05.874516 28875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19c6cb49e084d70a47d2b317fa0a46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:05.877185 28875 tablet_copy_client.cc:791] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: tablet copy: Starting download of 0 data blocks...
I20260925 02:26:05.877350 28875 tablet_copy_client.cc:655] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: tablet copy: Starting download of 1 WAL segments...
I20260925 02:26:05.878296 28875 tablet_copy_client.cc:531] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:26:05.878993 28875 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Bootstrap starting.
I20260925 02:26:05.880308 28875 log.cc:824] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:05.882730 28875 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:26:05.882964 28875 tablet_bootstrap.cc:492] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Bootstrap complete.
I20260925 02:26:05.883104 28875 ts_tablet_manager.cc:1397] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:26:05.884055 28875 raft_consensus.cc:348] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:05.884219 28875 raft_consensus.cc:729] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c876f1372ab848ce9394d0bcde654288, State: Initialized, Role: FOLLOWER
I20260925 02:26:05.884364 28875 consensus_queue.cc:260] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c876f1372ab848ce9394d0bcde654288" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45615 } } peers { permanent_uuid: "ad6f4954ea1043629b8b6733baf53868" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 46347 } } peers { permanent_uuid: "32c89f3d785841758a0bbb3ad440bcf1" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37741 } }
I20260925 02:26:05.884631 28875 ts_tablet_manager.cc:1428] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20260925 02:26:05.884964 28557 tablet_copy_service.cc:342] P 32c89f3d785841758a0bbb3ad440bcf1: Request end of tablet copy session c876f1372ab848ce9394d0bcde654288-d19c6cb49e084d70a47d2b317fa0a46d received from {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:34877
I20260925 02:26:05.885006 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.884846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:55391)
0925 02:26:05.884934 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.884974 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:26:05.885026 28557 tablet_copy_service.cc:433] P 32c89f3d785841758a0bbb3ad440bcf1: ending tablet copy session c876f1372ab848ce9394d0bcde654288-d19c6cb49e084d70a47d2b317fa0a46d on tablet d19c6cb49e084d70a47d2b317fa0a46d with peer c876f1372ab848ce9394d0bcde654288
I20260925 02:26:05.885303 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:05.885216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:47237)
0925 02:26:05.885258 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:05.885281 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:26:06.415625 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:06.415473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:41241)
0925 02:26:06.415551 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:06.415576 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64,"thread_start_us":38,"threads_started":1}
I20260925 02:26:06.415644 28811 raft_consensus.cc:1203] T d19c6cb49e084d70a47d2b317fa0a46d P c876f1372ab848ce9394d0bcde654288 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 02:26:06.416373 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:06.416140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:60649)
0925 02:26:06.416252 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:06.416296 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260925 02:26:06.460019 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:06.459771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:39915)
0925 02:26:06.459927 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:06.459970 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260925 02:26:07.416455 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:07.416196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:33511)
0925 02:26:07.416336 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:07.416403 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260925 02:26:07.417021 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:07.416760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:55053)
0925 02:26:07.416939 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:07.416983 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260925 02:26:07.460788 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:07.460543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:46447)
0925 02:26:07.460687 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:07.460736 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260925 02:26:08.417436 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:08.417067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:33465)
0925 02:26:08.417313 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:08.417384 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
I20260925 02:26:08.417730 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:08.417474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:48765)
0925 02:26:08.417633 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:08.417667 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260925 02:26:08.461790 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:08.461320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:48743)
0925 02:26:08.461680 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:08.461741 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":302,"threads_started":1}
I20260925 02:26:09.418231 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:09.417971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:34129)
0925 02:26:09.418138 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:09.418180 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260925 02:26:09.418465 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:09.418215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:35533)
0925 02:26:09.418376 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:09.418415 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 02:26:09.418690 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:09.418600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:40721)
0925 02:26:09.418636 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:09.418668 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 02:26:09.462460 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:09.462221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:33643)
0925 02:26:09.462371 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:09.462412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260925 02:26:10.419196 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:10.418930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:36037)
0925 02:26:10.419087 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:10.419130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260925 02:26:10.419404 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:10.419115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:33841)
0925 02:26:10.419255 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:10.419330 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260925 02:26:10.463207 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:10.462973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:53755)
0925 02:26:10.463111 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:10.463156 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 02:26:11.420120 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:11.419927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:46425)
0925 02:26:11.420048 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:11.420078 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260925 02:26:11.419978 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:11.419689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:49693)
0925 02:26:11.419888 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:11.419930 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
I20260925 02:26:11.464066 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:11.463814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:50149)
0925 02:26:11.463972 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:11.464016 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260925 02:26:12.421039 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:12.420760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:46743)
0925 02:26:12.420961 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:12.420995 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
I20260925 02:26:12.421197 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:12.420760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:57531)
0925 02:26:12.420944 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:12.420992 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260925 02:26:12.464828 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:12.464583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:37149)
0925 02:26:12.464733 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:12.464778 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260925 02:26:13.421842 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:13.421642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:47105)
0925 02:26:13.421739 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:13.421788 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260925 02:26:13.421854 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:13.421548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:51187)
0925 02:26:13.421755 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:13.421802 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260925 02:26:13.465634 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:13.465404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:43567)
0925 02:26:13.465542 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:13.465585 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 02:26:14.422565 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:14.422328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:53981)
0925 02:26:14.422468 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:14.422517 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260925 02:26:14.422587 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:14.422366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:35433)
0925 02:26:14.422510 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:14.422544 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260925 02:26:14.466419 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:14.466158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:51293)
0925 02:26:14.466326 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:14.466369 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260925 02:26:15.423529 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:15.423242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:37591)
0925 02:26:15.423410 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:15.423451 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260925 02:26:15.423560 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:15.423154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:51385)
0925 02:26:15.423437 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:15.423502 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":242,"threads_started":1}
I20260925 02:26:15.467108 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:15.466896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:50265)
0925 02:26:15.467028 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:15.467061 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260925 02:26:16.424345 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:16.424124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:59187)
0925 02:26:16.424264 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:16.424294 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260925 02:26:16.424367 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:16.424091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:55521)
0925 02:26:16.424255 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:16.424297 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260925 02:26:16.467873 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:16.467629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:39363)
0925 02:26:16.467775 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:16.467816 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260925 02:26:17.425414 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:17.425153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:40067)
0925 02:26:17.425300 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:17.425365 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260925 02:26:17.425475 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:17.425175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:45687)
0925 02:26:17.425326 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:17.425420 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":205,"threads_started":1}
I20260925 02:26:17.468578 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:17.468330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:37399)
0925 02:26:17.468485 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:17.468530 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260925 02:26:18.426332 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:18.426076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.1:46095)
0925 02:26:18.426248 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:18.426287 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260925 02:26:18.426328 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:18.426050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.3:48753)
0925 02:26:18.426224 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:18.426279 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260925 02:26:18.469434 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:18.469109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43335 (local address 127.11.125.2:39073)
0925 02:26:18.469317 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:18.469379 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.62:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260925 02:26:19.107513 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28417
I20260925 02:26:19.113224 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28589
I20260925 02:26:19.118221 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28727
2026-09-25T02:26:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58375 ms)
[----------] 30 tests from SecurityITest (236646 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:19 dist-test-slave-hj7s krb5kdc[28929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:19 dist-test-slave-hj7s krb5kdc[28929](info): set up 2 sockets
Sep 25 02:26:19 dist-test-slave-hj7s krb5kdc[28929](info): commencing operation
krb5kdc: starting...
W20260925 02:26:20.175127 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:20.295995 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44337
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:20.377249 28942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:20.377491 28942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:20.377516 28942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:20.378721 28942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:20.378757 28942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:20.378772 28942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:20.378784 28942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:20.378796 28942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:20.380199 28942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:20.380393 28942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:20.380570 28942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:20.385800 28947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.385870 28942 server_base.cc:1034] running on GCE node
W20260925 02:26:20.385912 28948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:20.385805 28950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.386292 28942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:20.386541 28942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:20.387706 28942 hybrid_clock.cc:648] HybridClock initialized: now 1790303180387695 us; error 45 us; skew 500 ppm
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:20.390201 28942 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:20.390686 28942 webserver.cc:460] Webserver started at http://127.11.125.62:37533/ using document root <none> and password file <none>
I20260925 02:26:20.390926 28942 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:20.390995 28942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:20.391149 28942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:20.392215 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "a7438db59e6a49e6913c8c333e6389f3"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "39a2126049d5afae55a0c05aae0d5b54"
server_key_iv: "a0784f8471f52a1113cd0b8e11372ca2"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.392628 28942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "a7438db59e6a49e6913c8c333e6389f3"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "39a2126049d5afae55a0c05aae0d5b54"
server_key_iv: "a0784f8471f52a1113cd0b8e11372ca2"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.394065 28942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:20.394831 28957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.395023 28942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:20.395126 28942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "a7438db59e6a49e6913c8c333e6389f3"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "39a2126049d5afae55a0c05aae0d5b54"
server_key_iv: "a0784f8471f52a1113cd0b8e11372ca2"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.395216 28942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:20.444707 28942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:20.446801 28942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:20.446982 28942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:20.451329 28942 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44337
I20260925 02:26:20.451411 29009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44337 every 8 connection(s)
I20260925 02:26:20.451807 28942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:20.452736 29010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:20.455060 29010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: Bootstrap starting.
I20260925 02:26:20.455662 29010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:20.455971 29010 log.cc:824] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:20.456691 29010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: No bootstrap required, opened a new log
I20260925 02:26:20.458321 29010 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } }
I20260925 02:26:20.458489 29010 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:20.458513 29010 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7438db59e6a49e6913c8c333e6389f3, State: Initialized, Role: FOLLOWER
I20260925 02:26:20.458693 29010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } }
I20260925 02:26:20.458765 29010 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:20.458786 29010 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:20.458854 29010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:20.459376 29010 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } }
I20260925 02:26:20.459491 29010 leader_election.cc:302] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7438db59e6a49e6913c8c333e6389f3; no voters:
I20260925 02:26:20.459682 29010 leader_election.cc:288] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:20.459744 29013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:20.459926 29013 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [term 1 LEADER]: Becoming Leader. State: Replica: a7438db59e6a49e6913c8c333e6389f3, State: Running, Role: LEADER
I20260925 02:26:20.459947 29010 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:20.460067 29013 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } }
I20260925 02:26:20.460470 29015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7438db59e6a49e6913c8c333e6389f3. Latest consensus state: current_term: 1 leader_uuid: "a7438db59e6a49e6913c8c333e6389f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } } }
I20260925 02:26:20.460587 29015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:20.460605 29014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7438db59e6a49e6913c8c333e6389f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7438db59e6a49e6913c8c333e6389f3" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44337 } } }
I20260925 02:26:20.460660 29014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:20.460851 29022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:20.461400 29022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:20.461431 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 28942
I20260925 02:26:20.461530 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:20.461730 11764 external_mini_cluster.cc:1428] Setting key 1388384a63ff85847f8aea708427717e
I20260925 02:26:20.463881 29022 catalog_manager.cc:1347] Generated new cluster ID: 66dc616ffcd74d9096415ea5c57a65b6
I20260925 02:26:20.463941 29022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:20.468452 29022 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:20.469033 29022 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:20.472496 29022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: Generated new TSK 0
I20260925 02:26:20.472676 29022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:20.474576 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.462230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.11.125.62:44337)
0925 02:26:20.462416 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.462425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.462490 (+ 65us) server_negotiation.cc:407] Connection header received
0925 02:26:20.462797 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.462806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.462873 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.463024 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:20.463938 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.465364 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.466461 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.467071 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.467177 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.470074 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:20.470079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:20.470079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:20.470094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:20.472811 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.473202 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.473205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.473206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.473323 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.473543 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.473544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.473545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.473846 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:20.473984 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.474076 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.474346 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:26:20.504964 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:20.587402 29036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:20.587631 29036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:20.587673 29036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:20.588871 29036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:20.588954 29036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:20.589010 29036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:20.590497 29036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:20.590770 29036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:20.591022 29036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:20.597052 29042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:20.597055 29041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:20.597043 29044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.597383 29036 server_base.cc:1034] running on GCE node
I20260925 02:26:20.597581 29036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:20.597826 29036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:20.598971 29036 hybrid_clock.cc:648] HybridClock initialized: now 1790303180598945 us; error 41 us; skew 500 ppm
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:20.601661 29036 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:26:20.602146 29036 webserver.cc:460] Webserver started at http://127.11.125.1:46361/ using document root <none> and password file <none>
I20260925 02:26:20.602355 29036 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:20.602406 29036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:20.602509 29036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:20.603746 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "897e22572c9e4172b02eb6045a9e8d73"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "ca75480fa3d0371fbbfbffafdc57175e"
server_key_iv: "3d38adecbaab469c5c8cc58afbe7fd2c"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.604110 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "897e22572c9e4172b02eb6045a9e8d73"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "ca75480fa3d0371fbbfbffafdc57175e"
server_key_iv: "3d38adecbaab469c5c8cc58afbe7fd2c"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.605726 29036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:20.606406 29051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.606575 29036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:20.606688 29036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "897e22572c9e4172b02eb6045a9e8d73"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "ca75480fa3d0371fbbfbffafdc57175e"
server_key_iv: "3d38adecbaab469c5c8cc58afbe7fd2c"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.606765 29036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:20.626756 29036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:20.628839 29036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:20.629036 29036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:20.629308 29036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:20.629637 29036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:20.629694 29036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.629770 29036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:20.629803 29036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.635761 29036 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37681
I20260925 02:26:20.635869 29166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37681 every 8 connection(s)
I20260925 02:26:20.636181 29036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:20.640386 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29036
I20260925 02:26:20.640535 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:20.640828 11764 external_mini_cluster.cc:1428] Setting key e05f622589fa1d3591d1d585f67d3d74
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:20.646759 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.637090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44337 (local address 127.11.125.1:46591)
0925 02:26:20.637318 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.637356 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.637842 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.638055 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.638058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.638635 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:20.638835 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.638841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.639489 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.639490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.640348 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.640350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.640840 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.640842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.641013 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.641761 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:20.641769 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:20.643160 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:20.645242 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.645249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.645251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.645591 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.645840 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.645841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.645841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.645945 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.646248 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:20.646253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:20.646387 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.646470 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.646590 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260925 02:26:20.646780 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.637115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:46591 (local address 127.11.125.62:44337)
0925 02:26:20.637193 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.637196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.637444 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:26:20.637934 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.637935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.637946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.637999 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:20.638916 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.639412 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.640458 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.640750 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.640829 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.643269 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:20.643271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:20.643271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:20.643279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:20.645059 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.645681 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.645683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.645683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.645763 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.646022 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.646024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.646024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.646179 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:20.646265 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.646527 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.646629 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:26:20.647289 29167 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44337
I20260925 02:26:20.647405 29167 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:20.647621 29167 heartbeater.cc:511] Master 127.11.125.62:44337 requested a full tablet report, sending...
I20260925 02:26:20.648211 28974 ts_manager.cc:194] Registered new tserver with Master: 897e22572c9e4172b02eb6045a9e8d73 (127.11.125.1:37681)
I20260925 02:26:20.648810 28974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:46591
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:26:20.668974 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:20.750569 29172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:20.750765 29172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:20.750788 29172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:20.751946 29172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:20.751988 29172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:20.752014 29172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:20.753559 29172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:20.753818 29172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:20.754060 29172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:20.759268 29177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.759320 29172 server_base.cc:1034] running on GCE node
W20260925 02:26:20.759280 29180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:20.759310 29178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.759792 29172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:20.760003 29172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:20.761142 29172 hybrid_clock.cc:648] HybridClock initialized: now 1790303180761129 us; error 28 us; skew 500 ppm
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:20.763882 29172 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:26:20.764281 29172 webserver.cc:460] Webserver started at http://127.11.125.2:45483/ using document root <none> and password file <none>
I20260925 02:26:20.764592 29172 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:20.764667 29172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:20.764824 29172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:20.766005 29172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "1ad06d74f36d451e95ff2e1ed4c33486"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "fbb212595583684f6cfc0cb210457c98"
server_key_iv: "3e60c3d9cf061bff062dcd56b9451322"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.766397 29172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "1ad06d74f36d451e95ff2e1ed4c33486"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "fbb212595583684f6cfc0cb210457c98"
server_key_iv: "3e60c3d9cf061bff062dcd56b9451322"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.767771 29172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:20.768514 29187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.768694 29172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:20.768805 29172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "1ad06d74f36d451e95ff2e1ed4c33486"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "fbb212595583684f6cfc0cb210457c98"
server_key_iv: "3e60c3d9cf061bff062dcd56b9451322"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.768882 29172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:20.778771 29172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:20.780579 29172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:20.780747 29172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:20.781019 29172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:20.781333 29172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:20.781395 29172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.781431 29172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:20.781458 29172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.787560 29172 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:37035
I20260925 02:26:20.787600 29301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:37035 every 8 connection(s)
I20260925 02:26:20.787936 29172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:20.793907 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29172
I20260925 02:26:20.794039 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:20.794240 11764 external_mini_cluster.cc:1428] Setting key d19838737fa9426546d626983a6f56b2
I20260925 02:26:20.797367 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.788706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:39061 (local address 127.11.125.62:44337)
0925 02:26:20.788763 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.788764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.789012 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:26:20.789503 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.789504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.789515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.789566 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:20.790463 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.790952 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.791992 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.792171 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.792223 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.794409 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:20.794411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:20.794412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:20.794419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:20.795838 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.796381 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.796382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.796382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.796441 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.796720 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.796721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.796721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.796819 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:20.796883 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.797239 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.797303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:26:20.797752 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.788680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44337 (local address 127.11.125.2:39061)
0925 02:26:20.788831 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.788874 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.789423 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.789623 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.789626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.790176 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:20.790383 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.790388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.791033 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.791033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.791916 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.791919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.792242 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.792243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.792344 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.792885 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:20.792893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:20.794301 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:20.795950 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.795957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.795959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.796296 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.796512 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.796513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.796513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.796655 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.796894 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:20.796924 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:20.797091 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.797219 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.797404 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260925 02:26:20.798277 29302 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44337
I20260925 02:26:20.798399 29302 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:20.798602 29302 heartbeater.cc:511] Master 127.11.125.62:44337 requested a full tablet report, sending...
I20260925 02:26:20.798985 28974 ts_manager.cc:194] Registered new tserver with Master: 1ad06d74f36d451e95ff2e1ed4c33486 (127.11.125.2:37035)
I20260925 02:26:20.799358 28974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:39061
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:26:20.818965 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:20.901714 29307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:20.901924 29307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:20.901948 29307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:20.903124 29307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:20.903167 29307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:20.903192 29307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:20.904641 29307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:44337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:20.904898 29307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:20.905191 29307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:20.910101 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:20.910060 29315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.910203 29307 server_base.cc:1034] running on GCE node
W20260925 02:26:20.910064 29312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:20.910561 29307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:20.910804 29307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:20.911976 29307 hybrid_clock.cc:648] HybridClock initialized: now 1790303180911959 us; error 46 us; skew 500 ppm
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:20.914453 29307 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:26:20.914894 29307 webserver.cc:460] Webserver started at http://127.11.125.3:45079/ using document root <none> and password file <none>
I20260925 02:26:20.915171 29307 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:20.915251 29307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:20.915438 29307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:20.916463 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "85d17b6f128e453c8622468d110a2d85"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "46d0aebf96551f6818a39b89d72150a2"
server_key_iv: "ef2aa3d766a887fe954752f93de74d7d"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.916813 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "85d17b6f128e453c8622468d110a2d85"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "46d0aebf96551f6818a39b89d72150a2"
server_key_iv: "ef2aa3d766a887fe954752f93de74d7d"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.918114 29307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:26:20.918856 29322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.919034 29307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:20.919138 29307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "85d17b6f128e453c8622468d110a2d85"
format_stamp: "Formatted at 2026-09-25 02:26:20 on dist-test-slave-hj7s"
server_key: "46d0aebf96551f6818a39b89d72150a2"
server_key_iv: "ef2aa3d766a887fe954752f93de74d7d"
server_key_version: "encryptionkey@0"
I20260925 02:26:20.919210 29307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:20.929453 29307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:20.931777 29307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:20.931954 29307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:20.932199 29307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:20.932525 29307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:20.932582 29307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.932626 29307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:20.932651 29307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:20.938808 29307 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:43605
I20260925 02:26:20.938853 29435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:43605 every 8 connection(s)
I20260925 02:26:20.939181 29307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:26:20.943598 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29307
I20260925 02:26:20.943729 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:20.943997 11764 external_mini_cluster.cc:1428] Setting key 6cfa8495bc7f35423289b1a3fd0b7a88
Sep 25 02:26:20 dist-test-slave-hj7s krb5kdc[28929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:20.948163 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.939931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:39601 (local address 127.11.125.62:44337)
0925 02:26:20.940006 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.940009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.940242 (+ 233us) server_negotiation.cc:407] Connection header received
0925 02:26:20.940710 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.940711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.940721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.940776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:20.941793 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.942286 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.943234 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.943447 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.943496 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.945853 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:20.945854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:20.945854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:20.945860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:20.946980 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.947357 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.947357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.947358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.947407 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.947623 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.947624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.947624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.947717 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:20.947766 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.948036 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.948100 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:26:20.948298 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.939915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:44337 (local address 127.11.125.3:39601)
0925 02:26:20.940114 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.940152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.940625 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.940856 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.940860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.941522 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:20.941716 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.941721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.942383 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.942384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.943159 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.943162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.943528 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.943530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.943694 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.944330 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:20.944343 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:20.945766 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:20.947050 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.947057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.947058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.947277 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.947467 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:20.947468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:20.947468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:20.947567 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:20.947770 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:20.947774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:20.947903 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.947984 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.948099 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260925 02:26:20.948737 29436 heartbeater.cc:347] Connected to a master server at 127.11.125.62:44337
I20260925 02:26:20.948836 29436 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:20.949076 29436 heartbeater.cc:511] Master 127.11.125.62:44337 requested a full tablet report, sending...
I20260925 02:26:20.949460 28974 ts_manager.cc:194] Registered new tserver with Master: 85d17b6f128e453c8622468d110a2d85 (127.11.125.3:43605)
I20260925 02:26:20.949800 28974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:39601
I20260925 02:26:20.955658 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:20.963567 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.957149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.11.125.62:44337)
0925 02:26:20.957211 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.957213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.957343 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:26:20.957381 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.957382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.957391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.957437 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:20.957987 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.958434 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.959172 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.959401 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.959457 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.960754 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:20.960756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:20.960756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:20.960762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:20.962527 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.962851 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.962851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.962852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.962911 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:20.963083 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:20.963084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:20.963084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:20.963237 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:20.963319 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.963412 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.963473 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:26:20.965407 28974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:20.966974 28974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:20.973664 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37035 (local address 127.0.0.1:37366)
0925 02:26:20.969752 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.969799 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.969884 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.970097 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.970100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.970103 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:20.970191 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.970193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971348 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.971349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.972405 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.972408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.973358 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.973360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.973441 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.973456 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.973533 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.973585 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":65,"threads_started":1}
I20260925 02:26:20.973678 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.11.125.2:37035)
0925 02:26:20.969760 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.969765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.969875 (+ 110us) server_negotiation.cc:407] Connection header received
0925 02:26:20.969976 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.969979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.969996 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.970079 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:20.970298 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971242 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.972497 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.973241 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.973316 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.973378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.973559 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.973601 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260925 02:26:20.974249 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:43605 (local address 127.0.0.1:54518)
0925 02:26:20.970116 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.970125 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.970173 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.970423 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.970424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.970426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:20.970462 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.970463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971841 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.971841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.972983 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.972984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.974016 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.974017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.974080 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.974085 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.974120 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.974163 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260925 02:26:20.974538 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37681 (local address 127.0.0.1:49126)
0925 02:26:20.969994 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:20.970003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:20.970049 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:20.970297 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:20.970297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:20.970299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:20.970360 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.970362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971748 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.971749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.972966 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:20.972970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.974317 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.974318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:20.974382 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.974386 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:20.974439 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:26:20.974477 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260925 02:26:20.974571 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.11.125.1:37681)
0925 02:26:20.969899 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.969903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.970169 (+ 266us) server_negotiation.cc:407] Connection header received
0925 02:26:20.970214 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.970217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.970232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.970302 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:20.970416 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971624 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.973072 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.974169 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.974228 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.974281 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.974455 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.974494 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":92,"threads_started":1}
I20260925 02:26:20.974478 29236 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 (DEFAULT_TABLE table=test-table [id=e2a2455d9a5e469eb99876ee69c07589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:20.974861 29236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcdc916a5fa44a4821bb4b9193f41e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:20.977345 29100 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 (DEFAULT_TABLE table=test-table [id=e2a2455d9a5e469eb99876ee69c07589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:20.977677 29100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcdc916a5fa44a4821bb4b9193f41e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:20.977584 29450 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Bootstrap starting.
I20260925 02:26:20.978235 29450 tablet_bootstrap.cc:654] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:20.978734 29450 log.cc:824] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:20.979444 29450 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: No bootstrap required, opened a new log
I20260925 02:26:20.979497 29450 ts_tablet_manager.cc:1397] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:20.980657 29452 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Bootstrap starting.
I20260925 02:26:20.980909 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:20.969540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.11.125.3:43605)
0925 02:26:20.969715 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:20.969720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:20.970191 (+ 471us) server_negotiation.cc:407] Connection header received
0925 02:26:20.970240 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:20.970242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:20.970260 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:20.970337 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:20.970545 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.971650 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.973073 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:20.973927 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:20.980670 (+ 6743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:20.980744 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:20.980775 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 02:26:20.980834 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260925 02:26:20.981362 29452 tablet_bootstrap.cc:654] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:20.981367 29450 raft_consensus.cc:348] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.981563 29450 raft_consensus.cc:374] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:20.981613 29450 raft_consensus.cc:729] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad06d74f36d451e95ff2e1ed4c33486, State: Initialized, Role: FOLLOWER
I20260925 02:26:20.981629 29370 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 (DEFAULT_TABLE table=test-table [id=e2a2455d9a5e469eb99876ee69c07589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:20.981747 29450 consensus_queue.cc:260] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.981937 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcdc916a5fa44a4821bb4b9193f41e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:20.981982 29452 log.cc:824] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:20.982045 29450 ts_tablet_manager.cc:1428] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:20.982185 29302 heartbeater.cc:503] Master 127.11.125.62:44337 was elected leader, sending a full tablet report...
I20260925 02:26:20.982757 29452 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: No bootstrap required, opened a new log
I20260925 02:26:20.982846 29452 ts_tablet_manager.cc:1397] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:26:20.984215 29452 raft_consensus.cc:348] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.984378 29452 raft_consensus.cc:374] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:20.984426 29452 raft_consensus.cc:729] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897e22572c9e4172b02eb6045a9e8d73, State: Initialized, Role: FOLLOWER
I20260925 02:26:20.984548 29452 consensus_queue.cc:260] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.984598 29455 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Bootstrap starting.
I20260925 02:26:20.984925 29167 heartbeater.cc:503] Master 127.11.125.62:44337 was elected leader, sending a full tablet report...
I20260925 02:26:20.984839 29452 ts_tablet_manager.cc:1428] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:26:20.985335 29455 tablet_bootstrap.cc:654] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:20.985932 29455 log.cc:824] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:20.986577 29455 tablet_bootstrap.cc:492] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: No bootstrap required, opened a new log
I20260925 02:26:20.986673 29455 ts_tablet_manager.cc:1397] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:20.987990 29455 raft_consensus.cc:348] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.988148 29455 raft_consensus.cc:374] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:20.988195 29455 raft_consensus.cc:729] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d17b6f128e453c8622468d110a2d85, State: Initialized, Role: FOLLOWER
I20260925 02:26:20.988337 29455 consensus_queue.cc:260] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:20.988632 29455 ts_tablet_manager.cc:1428] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:20.988662 29436 heartbeater.cc:503] Master 127.11.125.62:44337 was elected leader, sending a full tablet report...
W20260925 02:26:21.038555 29303 tablet.cc:2369] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:21.109475 29456 raft_consensus.cc:482] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:21.109597 29456 raft_consensus.cc:504] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:21.109984 29456 leader_election.cc:288] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ad06d74f36d451e95ff2e1ed4c33486 (127.11.125.2:37035), 85d17b6f128e453c8622468d110a2d85 (127.11.125.3:43605)
I20260925 02:26:21.114077 29449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.110321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:47871 (local address 127.11.125.2:37035)
0925 02:26:21.110391 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.110393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.110403 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:21.110463 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.110465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.110476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.110536 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:21.110754 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.111548 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113107 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.113693 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113773 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.113782 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.113962 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.114003 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:26:21.114077 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.110276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50927 (local address 127.11.125.3:43605)
0925 02:26:21.110333 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.110353 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.110465 (+ 112us) server_negotiation.cc:407] Connection header received
0925 02:26:21.110512 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.110513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.110523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.110573 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:21.110753 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.111524 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113109 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.113693 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113773 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.113782 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.113962 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.114004 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:26:21.114096 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.110163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37035 (local address 127.11.125.1:47871)
0925 02:26:21.110209 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.110224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.110292 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.110568 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.110569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.110572 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:21.110676 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.110677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.111625 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.111626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.113013 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.113016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.113781 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.113878 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.113882 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.113941 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.114002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":95}
I20260925 02:26:21.114102 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.110163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:43605 (local address 127.11.125.1:50927)
0925 02:26:21.110389 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.110398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.110460 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.110584 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.110584 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.110585 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:21.110676 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.110677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.111625 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.111626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.113013 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.113016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.113781 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.113782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.113879 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.113883 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.113941 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.114002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":114,"thread_start_us":89,"threads_started":1}
I20260925 02:26:21.114492 29256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcdc916a5fa44a4821bb4b9193f41e2" candidate_uuid: "897e22572c9e4172b02eb6045a9e8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" is_pre_election: true
I20260925 02:26:21.114621 29256 raft_consensus.cc:2454] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897e22572c9e4172b02eb6045a9e8d73 in term 0.
I20260925 02:26:21.114852 29055 leader_election.cc:302] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ad06d74f36d451e95ff2e1ed4c33486, 897e22572c9e4172b02eb6045a9e8d73; no voters:
I20260925 02:26:21.114849 29390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcdc916a5fa44a4821bb4b9193f41e2" candidate_uuid: "897e22572c9e4172b02eb6045a9e8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d17b6f128e453c8622468d110a2d85" is_pre_election: true
I20260925 02:26:21.114984 29390 raft_consensus.cc:2454] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897e22572c9e4172b02eb6045a9e8d73 in term 0.
I20260925 02:26:21.115061 29456 raft_consensus.cc:2781] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:21.115149 29456 raft_consensus.cc:482] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:21.115192 29456 raft_consensus.cc:3037] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:21.115926 29456 raft_consensus.cc:504] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:21.116072 29456 leader_election.cc:288] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [CANDIDATE]: Term 1 election: Requested vote from peers 1ad06d74f36d451e95ff2e1ed4c33486 (127.11.125.2:37035), 85d17b6f128e453c8622468d110a2d85 (127.11.125.3:43605)
I20260925 02:26:21.116206 29256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcdc916a5fa44a4821bb4b9193f41e2" candidate_uuid: "897e22572c9e4172b02eb6045a9e8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ad06d74f36d451e95ff2e1ed4c33486"
I20260925 02:26:21.116273 29390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcdc916a5fa44a4821bb4b9193f41e2" candidate_uuid: "897e22572c9e4172b02eb6045a9e8d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d17b6f128e453c8622468d110a2d85"
I20260925 02:26:21.116308 29256 raft_consensus.cc:3037] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:21.116377 29390 raft_consensus.cc:3037] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:21.117097 29256 raft_consensus.cc:2454] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897e22572c9e4172b02eb6045a9e8d73 in term 1.
I20260925 02:26:21.117147 29390 raft_consensus.cc:2454] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897e22572c9e4172b02eb6045a9e8d73 in term 1.
I20260925 02:26:21.117301 29055 leader_election.cc:302] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ad06d74f36d451e95ff2e1ed4c33486, 897e22572c9e4172b02eb6045a9e8d73; no voters:
I20260925 02:26:21.117393 29456 raft_consensus.cc:2781] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:21.117532 29456 raft_consensus.cc:686] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [term 1 LEADER]: Becoming Leader. State: Replica: 897e22572c9e4172b02eb6045a9e8d73, State: Running, Role: LEADER
I20260925 02:26:21.117627 29456 consensus_queue.cc:237] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } }
I20260925 02:26:21.118573 28974 catalog_manager.cc:5495] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 897e22572c9e4172b02eb6045a9e8d73 (127.11.125.1). New cstate: current_term: 1 leader_uuid: "897e22572c9e4172b02eb6045a9e8d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897e22572c9e4172b02eb6045a9e8d73" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 } health_report { overall_health: UNKNOWN } } }
W20260925 02:26:21.137048 29168 tablet.cc:2369] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:21.153534 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.150907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.11.125.1:37681)
0925 02:26:21.150978 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.150980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.150991 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:21.151105 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.151107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.151129 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.151212 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:21.151384 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.151873 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.152677 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.152896 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.152993 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.153167 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:21.153284 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:21.153379 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.153402 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.153422 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:26:21.155288 29256 raft_consensus.cc:1261] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 1 FOLLOWER]: Refusing update from remote peer 897e22572c9e4172b02eb6045a9e8d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:21.155339 29390 raft_consensus.cc:1261] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 1 FOLLOWER]: Refusing update from remote peer 897e22572c9e4172b02eb6045a9e8d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:21.155691 29456 consensus_queue.cc:1035] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad06d74f36d451e95ff2e1ed4c33486" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:21.155884 29461 consensus_queue.cc:1035] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 897e22572c9e4172b02eb6045a9e8d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d17b6f128e453c8622468d110a2d85" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:21.158705 29464 mvcc.cc:205] Tried to move back new op lower bound from 7333081830009466880 to 7333081829858684928. Current Snapshot: MvccSnapshot[applied={T|T < 7333081830009466880}]
I20260925 02:26:21.159008 29465 mvcc.cc:205] Tried to move back new op lower bound from 7333081830009466880 to 7333081829858684928. Current Snapshot: MvccSnapshot[applied={T|T < 7333081830009466880}]
I20260925 02:26:21.159401 29466 mvcc.cc:205] Tried to move back new op lower bound from 7333081830009466880 to 7333081829858684928. Current Snapshot: MvccSnapshot[applied={T|T < 7333081830009466880}]
I20260925 02:26:21.161470 28974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:21.161566 28974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:21.162410 28974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7438db59e6a49e6913c8c333e6389f3: Sending DeleteTablet for 3 replicas of tablet 0dcdc916a5fa44a4821bb4b9193f41e2
I20260925 02:26:21.162760 29100 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:21 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:49126
I20260925 02:26:21.162765 29236 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:21 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:37366
I20260925 02:26:21.162907 29370 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcdc916a5fa44a4821bb4b9193f41e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:21 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:54518
I20260925 02:26:21.163117 29473 tablet_replica.cc:330] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: stopping tablet replica
I20260925 02:26:21.163151 29475 tablet_replica.cc:330] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: stopping tablet replica
I20260925 02:26:21.163255 29475 raft_consensus.cc:2229] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:21.163281 29473 raft_consensus.cc:2229] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:21.163368 29475 raft_consensus.cc:2258] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:21.163489 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29036
I20260925 02:26:21.163628 29473 raft_consensus.cc:2258] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:21.164412 29473 ts_tablet_manager.cc:1905] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:21.165117 29475 ts_tablet_manager.cc:1905] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:21.166649 29473 ts_tablet_manager.cc:1918] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:21.166752 29473 log.cc:1196] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/0dcdc916a5fa44a4821bb4b9193f41e2
I20260925 02:26:21.166883 29475 ts_tablet_manager.cc:1918] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:21.166986 29475 log.cc:1196] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/0dcdc916a5fa44a4821bb4b9193f41e2
I20260925 02:26:21.167039 29473 ts_tablet_manager.cc:1939] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 1ad06d74f36d451e95ff2e1ed4c33486: Deleting consensus metadata
I20260925 02:26:21.167263 29475 ts_tablet_manager.cc:1939] T 0dcdc916a5fa44a4821bb4b9193f41e2 P 85d17b6f128e453c8622468d110a2d85: Deleting consensus metadata
I20260925 02:26:21.167457 28961 catalog_manager.cc:4841] TS 1ad06d74f36d451e95ff2e1ed4c33486 (127.11.125.2:37035): tablet 0dcdc916a5fa44a4821bb4b9193f41e2 (table test-table [id=e2a2455d9a5e469eb99876ee69c07589]) successfully deleted
I20260925 02:26:21.167672 28958 catalog_manager.cc:4841] TS 85d17b6f128e453c8622468d110a2d85 (127.11.125.3:43605): tablet 0dcdc916a5fa44a4821bb4b9193f41e2 (table test-table [id=e2a2455d9a5e469eb99876ee69c07589]) successfully deleted
W20260925 02:26:21.170014 28959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:37681 (error 108)
I20260925 02:26:21.170552 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.170408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37681 (local address 127.0.0.1:49156)
0925 02:26:21.170470 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.170515 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:26:21.170560 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29172
W20260925 02:26:21.171581 28959 catalog_manager.cc:4584] TS 897e22572c9e4172b02eb6045a9e8d73 (127.11.125.1:37681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dcdc916a5fa44a4821bb4b9193f41e2: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37681: connect: Connection refused (error 111)
I20260925 02:26:21.175930 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29307
I20260925 02:26:21.180609 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 28942
2026-09-25T02:26:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:21Z Disabled control of system clock
I20260925 02:26:21.208948 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34233
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:21.289252 29480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:21.289455 29480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:21.289475 29480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:21.290534 29480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:21.290567 29480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:21.290580 29480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:21.290592 29480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:21.290604 29480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:21.291903 29480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:34233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:21.292092 29480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:21.292318 29480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:26:21.297238 29480 server_base.cc:1034] running on GCE node
W20260925 02:26:21.297173 29486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.297209 29485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.297196 29488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.297715 29480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:21.298036 29480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:21.299206 29480 hybrid_clock.cc:648] HybridClock initialized: now 1790303181299188 us; error 41 us; skew 500 ppm
I20260925 02:26:21.300498 29480 webserver.cc:460] Webserver started at http://127.11.125.62:34099/ using document root <none> and password file <none>
I20260925 02:26:21.300768 29480 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:21.300844 29480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:21.301017 29480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:21.302090 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "533c36a4bbc44b45887df050edfe4b4c"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "28c37239a2f72b8244bd4b7d6ac6c3ae"
server_key_iv: "4b078bafdc1b8c62aa3af0a43dc6c025"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.302462 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "533c36a4bbc44b45887df050edfe4b4c"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "28c37239a2f72b8244bd4b7d6ac6c3ae"
server_key_iv: "4b078bafdc1b8c62aa3af0a43dc6c025"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.303758 29480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:21.304477 29494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.304652 29480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:21.304746 29480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "533c36a4bbc44b45887df050edfe4b4c"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "28c37239a2f72b8244bd4b7d6ac6c3ae"
server_key_iv: "4b078bafdc1b8c62aa3af0a43dc6c025"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.304833 29480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:21.318102 29480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:21.318440 29480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:21.318598 29480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:21.322896 29480 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:34233
I20260925 02:26:21.322928 29547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:34233 every 8 connection(s)
I20260925 02:26:21.323328 29480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:21.324293 29548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:21.326856 29548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: Bootstrap starting.
I20260925 02:26:21.327440 29548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:21.327739 29548 log.cc:824] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:21.328415 29548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: No bootstrap required, opened a new log
I20260925 02:26:21.330031 29548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } }
I20260925 02:26:21.330196 29548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:21.330245 29548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533c36a4bbc44b45887df050edfe4b4c, State: Initialized, Role: FOLLOWER
I20260925 02:26:21.330432 29548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } }
I20260925 02:26:21.330535 29548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:21.330582 29548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:21.330633 29548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:21.331138 29548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } }
I20260925 02:26:21.331259 29548 leader_election.cc:302] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 533c36a4bbc44b45887df050edfe4b4c; no voters:
I20260925 02:26:21.331454 29548 leader_election.cc:288] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:21.331512 29551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:21.331717 29548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:21.331722 29551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [term 1 LEADER]: Becoming Leader. State: Replica: 533c36a4bbc44b45887df050edfe4b4c, State: Running, Role: LEADER
I20260925 02:26:21.331920 29551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } }
I20260925 02:26:21.332402 29552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "533c36a4bbc44b45887df050edfe4b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } } }
I20260925 02:26:21.332495 29552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:21.332715 29560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:21.332921 29553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533c36a4bbc44b45887df050edfe4b4c. Latest consensus state: current_term: 1 leader_uuid: "533c36a4bbc44b45887df050edfe4b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533c36a4bbc44b45887df050edfe4b4c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34233 } } }
I20260925 02:26:21.333253 29553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:21.333257 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29480
I20260925 02:26:21.333307 29560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:21.333369 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:21.333582 11764 external_mini_cluster.cc:1428] Setting key 02e9581388dd01a86e97615740ece984
I20260925 02:26:21.338581 29560 catalog_manager.cc:1347] Generated new cluster ID: e500a7c190f14857a57db697149f9e7a
I20260925 02:26:21.338639 29560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:21.339080 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.334238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.11.125.62:34233)
0925 02:26:21.334468 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.334475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.334498 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:26:21.334585 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.334590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.334649 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.334785 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:21.335026 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.335912 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.336700 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.337189 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.337321 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.338064 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:21.338068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:21.338068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:21.338294 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:21.338337 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:21.338352 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:21.338420 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.338549 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.338707 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260925 02:26:21.352252 29560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:21.352773 29560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:21.359227 29560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: Generated new TSK 0
I20260925 02:26:21.359419 29560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:21.363885 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--builtin_ntp_servers=127.11.125.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:21.460496 29573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:21.460700 29573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:21.460722 29573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:21.461946 29573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:21.461990 29573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:21.462016 29573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:21.463399 29573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:21.463646 29573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:21.463976 29573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:21.469062 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.469259 29573 server_base.cc:1034] running on GCE node
W20260925 02:26:21.469067 29581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.469067 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.469619 29573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:21.469902 29573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:21.471084 29573 hybrid_clock.cc:648] HybridClock initialized: now 1790303181471056 us; error 42 us; skew 500 ppm
I20260925 02:26:21.472693 29573 webserver.cc:460] Webserver started at http://127.11.125.1:44203/ using document root <none> and password file <none>
I20260925 02:26:21.473016 29573 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:21.473083 29573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:21.473234 29573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:21.474720 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "8012d58c3806433b9b8127dc2140ba4a"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "1c951eaf7fdcf8539af38380ab8c941f"
server_key_iv: "8246ab6506559bc45ab1092089f08552"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.475175 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "8012d58c3806433b9b8127dc2140ba4a"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "1c951eaf7fdcf8539af38380ab8c941f"
server_key_iv: "8246ab6506559bc45ab1092089f08552"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.477090 29573 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:26:21.478061 29587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.478278 29573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:21.478381 29573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "8012d58c3806433b9b8127dc2140ba4a"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "1c951eaf7fdcf8539af38380ab8c941f"
server_key_iv: "8246ab6506559bc45ab1092089f08552"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.478482 29573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:21.550689 29573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:21.551016 29573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:21.551132 29573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:21.551400 29573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:21.551803 29573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:21.551841 29573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.551867 29573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:21.551882 29573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.558349 29573 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:38811
I20260925 02:26:21.558406 29701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:38811 every 8 connection(s)
I20260925 02:26:21.558770 29573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:21.560614 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29573
I20260925 02:26:21.560726 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:21.560973 11764 external_mini_cluster.cc:1428] Setting key 36bf348555f6d279b0d9a9aa81a6be35
I20260925 02:26:21.562049 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--builtin_ntp_servers=127.11.125.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:26:21.564805 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.559541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:56697 (local address 127.11.125.62:34233)
0925 02:26:21.559613 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.559615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.559920 (+ 305us) server_negotiation.cc:407] Connection header received
0925 02:26:21.560381 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.560383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.560398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.560466 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:21.561505 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.562154 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.563195 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.563413 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.563505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.564419 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:21.564420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:21.564421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:21.564426 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:21.564447 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:21.564452 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:21.564499 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.564643 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.564679 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:26:21.565003 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.559533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:34233 (local address 127.11.125.1:56697)
0925 02:26:21.559742 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.559785 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.560288 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.560554 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.560559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.560573 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:21.561398 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.561405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.562286 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.562287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.563089 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.563092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.563518 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.563519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.563660 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.564257 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:21.564257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:21.564288 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:21.564313 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:21.564526 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:21.564534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:21.564537 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.564610 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.564729 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260925 02:26:21.565474 29702 heartbeater.cc:347] Connected to a master server at 127.11.125.62:34233
I20260925 02:26:21.565570 29702 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:21.565779 29702 heartbeater.cc:511] Master 127.11.125.62:34233 requested a full tablet report, sending...
I20260925 02:26:21.566606 29512 ts_manager.cc:194] Registered new tserver with Master: 8012d58c3806433b9b8127dc2140ba4a (127.11.125.1:38811)
I20260925 02:26:21.567749 29512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.1:56697
W20260925 02:26:21.650043 29705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:21.650283 29705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:21.650309 29705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:21.651628 29705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:21.651679 29705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:21.651741 29705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:21.653609 29705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:21.653877 29705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:21.654286 29705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:21.660626 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.660673 29705 server_base.cc:1034] running on GCE node
W20260925 02:26:21.660647 29713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.660804 29710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.661079 29705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:21.661414 29705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:21.662640 29705 hybrid_clock.cc:648] HybridClock initialized: now 1790303181662611 us; error 33 us; skew 500 ppm
I20260925 02:26:21.664013 29705 webserver.cc:460] Webserver started at http://127.11.125.2:33321/ using document root <none> and password file <none>
I20260925 02:26:21.664223 29705 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:21.664265 29705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:21.664371 29705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:21.665597 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "d7ba9837773544f89ba3d1081c030b9d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "51ef39ba657ded36130ae680d1a5398a"
server_key_iv: "bce931ab75e105fb391d3d795a2816c7"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.665915 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "d7ba9837773544f89ba3d1081c030b9d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "51ef39ba657ded36130ae680d1a5398a"
server_key_iv: "bce931ab75e105fb391d3d795a2816c7"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.667282 29705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:21.668327 29719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.668546 29705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:21.668645 29705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "d7ba9837773544f89ba3d1081c030b9d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "51ef39ba657ded36130ae680d1a5398a"
server_key_iv: "bce931ab75e105fb391d3d795a2816c7"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.668705 29705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:21.706652 29705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:21.707029 29705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:21.707185 29705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:21.707520 29705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:21.707935 29705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:21.707971 29705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.707995 29705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:21.708030 29705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.714041 29705 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:33691
I20260925 02:26:21.714135 29833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:33691 every 8 connection(s)
I20260925 02:26:21.714474 29705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:26:21.718092 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29705
I20260925 02:26:21.718192 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:21.718394 11764 external_mini_cluster.cc:1428] Setting key 7bc513904f57c71c3920ccaafb8f13a0
I20260925 02:26:21.719525 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--builtin_ntp_servers=127.11.125.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:26:21.720261 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.715256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:36907 (local address 127.11.125.62:34233)
0925 02:26:21.715343 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.715347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.715568 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:26:21.716071 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.716072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.716083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.716137 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:21.716808 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.717389 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.718678 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.718943 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.719016 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.720003 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:21.720004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:21.720005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:21.720009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:21.720022 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:21.720026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:21.720080 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.720176 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.720202 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:26:21.720556 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.715220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:34233 (local address 127.11.125.2:36907)
0925 02:26:21.715448 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.715483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.715982 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.716194 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.716197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.716205 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:21.716730 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.716734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.717475 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.717476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.718593 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.718596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.719041 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.719043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.719214 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.719824 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:21.719825 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:21.719876 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:21.719907 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:21.720096 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:21.720100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:21.720103 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.720173 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.720288 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":118,"threads_started":1}
I20260925 02:26:21.721319 29834 heartbeater.cc:347] Connected to a master server at 127.11.125.62:34233
I20260925 02:26:21.721428 29834 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:21.721632 29834 heartbeater.cc:511] Master 127.11.125.62:34233 requested a full tablet report, sending...
I20260925 02:26:21.722044 29512 ts_manager.cc:194] Registered new tserver with Master: d7ba9837773544f89ba3d1081c030b9d (127.11.125.2:33691)
I20260925 02:26:21.722503 29512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.2:36907
W20260925 02:26:21.803234 29837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:21.803464 29837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:21.803501 29837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:21.804625 29837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:21.804687 29837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:21.804739 29837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:21.806226 29837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:21.806499 29837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:21.806802 29837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:21.812145 29843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.812117 29842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:21.812117 29845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:21.812211 29837 server_base.cc:1034] running on GCE node
I20260925 02:26:21.812520 29837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:21.812784 29837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:21.813926 29837 hybrid_clock.cc:648] HybridClock initialized: now 1790303181813926 us; error 57 us; skew 500 ppm
I20260925 02:26:21.815198 29837 webserver.cc:460] Webserver started at http://127.11.125.3:43269/ using document root <none> and password file <none>
I20260925 02:26:21.815459 29837 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:21.815506 29837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:21.815697 29837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:21.816788 29837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "575473cf35164e2c9c383af36f3cea8d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "faf28e893ad93a8f25793b0b03de96fb"
server_key_iv: "386970578004f62caeec0990851a9701"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.817232 29837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "575473cf35164e2c9c383af36f3cea8d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "faf28e893ad93a8f25793b0b03de96fb"
server_key_iv: "386970578004f62caeec0990851a9701"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.818526 29837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:21.819286 29851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.819463 29837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:21.819572 29837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "575473cf35164e2c9c383af36f3cea8d"
format_stamp: "Formatted at 2026-09-25 02:26:21 on dist-test-slave-hj7s"
server_key: "faf28e893ad93a8f25793b0b03de96fb"
server_key_iv: "386970578004f62caeec0990851a9701"
server_key_version: "encryptionkey@0"
I20260925 02:26:21.819653 29837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:21.838595 29837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:21.838927 29837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:21.839083 29837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:21.839316 29837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:21.839666 29837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:21.839701 29837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.839725 29837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:21.839769 29837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:21.845599 29837 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:37879
I20260925 02:26:21.845692 29965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:37879 every 8 connection(s)
I20260925 02:26:21.846009 29837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:26:21.851923 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.846849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:51417 (local address 127.11.125.62:34233)
0925 02:26:21.846949 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.846951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.847209 (+ 258us) server_negotiation.cc:407] Connection header received
0925 02:26:21.847783 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.847784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.847798 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.847861 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:21.848600 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.849265 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.850354 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.850571 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.850695 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.851544 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:21.851546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:21.851546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:21.851553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:21.851569 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:21.851573 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:21.851631 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.851800 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.851835 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:26:21.852175 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.846763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:34233 (local address 127.11.125.3:51417)
0925 02:26:21.847017 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.847057 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.847655 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.847949 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.847953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.847962 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:21.848499 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.848503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.849372 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.849373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.850242 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.850245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.850713 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.850714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.850859 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.851390 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:21.851390 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:21.851420 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:21.851444 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:21.851638 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:21.851643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:21.851646 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.851800 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.851911 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
I20260925 02:26:21.852689 29966 heartbeater.cc:347] Connected to a master server at 127.11.125.62:34233
I20260925 02:26:21.852804 29966 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:21.853106 29966 heartbeater.cc:511] Master 127.11.125.62:34233 requested a full tablet report, sending...
I20260925 02:26:21.853569 29512 ts_manager.cc:194] Registered new tserver with Master: 575473cf35164e2c9c383af36f3cea8d (127.11.125.3:37879)
I20260925 02:26:21.853932 29512 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.3:51417
I20260925 02:26:21.855638 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 29837
I20260925 02:26:21.855741 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:21.855957 11764 external_mini_cluster.cc:1428] Setting key d0d8a4a310f310a50f53112129f4bcd1
I20260925 02:26:21.857033 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:21.861593 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.858484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.11.125.62:34233)
0925 02:26:21.858580 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.858582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.858734 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:26:21.858823 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.858825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.858838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.858887 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:21.859099 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.859671 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.860467 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.860637 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.860686 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.861342 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:21.861343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:21.861344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:21.861348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:21.861360 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:21.861364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:21.861441 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.861519 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.861539 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:26:21.863440 29512 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:21.865075 29512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:21.871778 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.11.125.3:37879)
0925 02:26:21.868049 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.868053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.868063 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:21.868108 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.868110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.868122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.868211 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:21.868325 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869190 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.870664 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871347 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.871446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.871544 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.871647 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.871686 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260925 02:26:21.871784 29979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:37879 (local address 127.0.0.1:39444)
0925 02:26:21.867777 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.867786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.867822 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.868209 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.868210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.868213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:21.868262 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.868264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869280 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.869281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.870564 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.870567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871446 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.871447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.871541 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.871557 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.871632 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.871699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260925 02:26:21.871791 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:33691 (local address 127.0.0.1:45780)
0925 02:26:21.867565 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.867601 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.867668 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.868048 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.868051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.868057 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:21.868149 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.868151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869603 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.869604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.870665 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.870667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871532 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.871533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.871605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.871609 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.871654 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.871699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":11,"thread_start_us":59,"threads_started":1}
I20260925 02:26:21.871946 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.11.125.1:38811)
0925 02:26:21.867850 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.867855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.867866 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:21.867911 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.867913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.867929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.868299 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:21.868375 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869545 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.870632 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871386 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.871690 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.871790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.871816 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.871858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260925 02:26:21.872120 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38811 (local address 127.0.0.1:43288)
0925 02:26:21.867557 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:21.867600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:21.867668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:21.868048 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:21.868051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:21.868054 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:21.868146 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.868148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869646 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.869647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.870548 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:21.870551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871534 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.871535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:21.871608 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.871610 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:26:21.871641 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 02:26:21.872068 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":384,"thread_start_us":94,"threads_started":1}
I20260925 02:26:21.872315 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:21.867666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.11.125.2:33691)
0925 02:26:21.867875 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:21.867879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:21.867889 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:21.867927 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:21.867929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:21.867943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:21.868227 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:21.868418 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.869510 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.870748 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:21.871439 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:21.872038 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:21.872129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:21.872150 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 02:26:21.872213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260925 02:26:21.872928 29635 tablet_service.cc:1467] Processing CreateTablet for tablet 768f25c7bc3648048d5a54275d43d849 (DEFAULT_TABLE table=test-table [id=fc1aec4118764b3699254a30602dba8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:21.872987 29767 tablet_service.cc:1467] Processing CreateTablet for tablet 768f25c7bc3648048d5a54275d43d849 (DEFAULT_TABLE table=test-table [id=fc1aec4118764b3699254a30602dba8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:21.873325 29635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768f25c7bc3648048d5a54275d43d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:21.873333 29767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768f25c7bc3648048d5a54275d43d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:21.873243 29899 tablet_service.cc:1467] Processing CreateTablet for tablet 768f25c7bc3648048d5a54275d43d849 (DEFAULT_TABLE table=test-table [id=fc1aec4118764b3699254a30602dba8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:21.873548 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768f25c7bc3648048d5a54275d43d849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:21.875936 29983 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Bootstrap starting.
I20260925 02:26:21.876535 29983 tablet_bootstrap.cc:654] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:21.876727 29985 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Bootstrap starting.
I20260925 02:26:21.876884 29983 log.cc:824] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:21.877307 29984 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Bootstrap starting.
I20260925 02:26:21.877447 29985 tablet_bootstrap.cc:654] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:21.877713 29983 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: No bootstrap required, opened a new log
I20260925 02:26:21.877799 29983 ts_tablet_manager.cc:1397] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:21.877791 29985 log.cc:824] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:21.877990 29984 tablet_bootstrap.cc:654] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:21.878326 29984 log.cc:824] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:21.878571 29985 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: No bootstrap required, opened a new log
I20260925 02:26:21.878644 29985 ts_tablet_manager.cc:1397] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:21.879159 29984 tablet_bootstrap.cc:492] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: No bootstrap required, opened a new log
I20260925 02:26:21.879278 29984 ts_tablet_manager.cc:1397] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:21.880125 29983 raft_consensus.cc:348] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.880285 29983 raft_consensus.cc:374] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:21.880338 29983 raft_consensus.cc:729] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ba9837773544f89ba3d1081c030b9d, State: Initialized, Role: FOLLOWER
I20260925 02:26:21.880470 29983 consensus_queue.cc:260] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.880740 29983 ts_tablet_manager.cc:1428] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:21.880889 29834 heartbeater.cc:503] Master 127.11.125.62:34233 was elected leader, sending a full tablet report...
I20260925 02:26:21.881317 29985 raft_consensus.cc:348] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.881487 29985 raft_consensus.cc:374] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:21.881525 29985 raft_consensus.cc:729] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575473cf35164e2c9c383af36f3cea8d, State: Initialized, Role: FOLLOWER
I20260925 02:26:21.881645 29985 consensus_queue.cc:260] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.881915 29985 ts_tablet_manager.cc:1428] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:26:21.881878 29984 raft_consensus.cc:348] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.882035 29966 heartbeater.cc:503] Master 127.11.125.62:34233 was elected leader, sending a full tablet report...
I20260925 02:26:21.882041 29984 raft_consensus.cc:374] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:21.882078 29984 raft_consensus.cc:729] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012d58c3806433b9b8127dc2140ba4a, State: Initialized, Role: FOLLOWER
I20260925 02:26:21.882192 29984 consensus_queue.cc:260] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:21.882467 29984 ts_tablet_manager.cc:1428] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:21.882517 29702 heartbeater.cc:503] Master 127.11.125.62:34233 was elected leader, sending a full tablet report...
W20260925 02:26:21.965171 29835 tablet.cc:2369] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:22.059489 29703 tablet.cc:2369] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:22.096657 29967 tablet.cc:2369] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:22.115530 29990 raft_consensus.cc:482] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:22.115712 29990 raft_consensus.cc:504] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:22.116132 29990 leader_election.cc:288] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ba9837773544f89ba3d1081c030b9d (127.11.125.2:33691), 8012d58c3806433b9b8127dc2140ba4a (127.11.125.1:38811)
I20260925 02:26:22.119114 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.116414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:60435 (local address 127.11.125.2:33691)
0925 02:26:22.116483 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:22.116486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:22.116516 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:26:22.116550 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:22.116551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:22.116564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:22.116622 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:22.116827 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.117389 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118381 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.118747 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118834 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:22.118843 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:22.118992 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:26:22.119036 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:26:22.119131 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.116486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:60483 (local address 127.11.125.1:38811)
0925 02:26:22.116557 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:22.116560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:22.116710 (+ 150us) server_negotiation.cc:407] Connection header received
0925 02:26:22.116741 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:22.116742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:22.116754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:22.116810 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:22.116991 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.117484 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118413 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.118748 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118835 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:22.118843 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:22.119025 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:26:22.119068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:26:22.119122 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.116325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:33691 (local address 127.11.125.3:60435)
0925 02:26:22.116395 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:22.116411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:22.116465 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:22.116628 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:22.116629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:22.116631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:22.116724 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:22.116727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.117475 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.117476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:22.118264 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:22.118266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.118834 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:22.118917 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:22.118921 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:22.118963 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 02:26:22.119027 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260925 02:26:22.119138 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.116329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38811 (local address 127.11.125.3:60483)
0925 02:26:22.116621 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:22.116633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:22.116696 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:22.116827 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:22.116827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:22.116829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:22.116929 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:22.116932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.117570 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.117571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:22.118287 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:22.118289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.118846 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.118848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:22.118928 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:22.118931 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:22.119003 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:26:22.119041 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":107,"thread_start_us":103,"threads_started":1}
I20260925 02:26:22.119496 29788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768f25c7bc3648048d5a54275d43d849" candidate_uuid: "575473cf35164e2c9c383af36f3cea8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ba9837773544f89ba3d1081c030b9d" is_pre_election: true
I20260925 02:26:22.119494 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768f25c7bc3648048d5a54275d43d849" candidate_uuid: "575473cf35164e2c9c383af36f3cea8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012d58c3806433b9b8127dc2140ba4a" is_pre_election: true
I20260925 02:26:22.119656 29788 raft_consensus.cc:2454] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575473cf35164e2c9c383af36f3cea8d in term 0.
I20260925 02:26:22.119656 29656 raft_consensus.cc:2454] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575473cf35164e2c9c383af36f3cea8d in term 0.
I20260925 02:26:22.119910 29852 leader_election.cc:302] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575473cf35164e2c9c383af36f3cea8d, 8012d58c3806433b9b8127dc2140ba4a; no voters:
I20260925 02:26:22.120065 29990 raft_consensus.cc:2781] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:22.120143 29990 raft_consensus.cc:482] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:22.120162 29990 raft_consensus.cc:3037] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:22.120893 29990 raft_consensus.cc:504] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:22.121064 29990 leader_election.cc:288] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [CANDIDATE]: Term 1 election: Requested vote from peers d7ba9837773544f89ba3d1081c030b9d (127.11.125.2:33691), 8012d58c3806433b9b8127dc2140ba4a (127.11.125.1:38811)
I20260925 02:26:22.121222 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768f25c7bc3648048d5a54275d43d849" candidate_uuid: "575473cf35164e2c9c383af36f3cea8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012d58c3806433b9b8127dc2140ba4a"
I20260925 02:26:22.121279 29788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768f25c7bc3648048d5a54275d43d849" candidate_uuid: "575473cf35164e2c9c383af36f3cea8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ba9837773544f89ba3d1081c030b9d"
I20260925 02:26:22.121335 29656 raft_consensus.cc:3037] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:22.121342 29788 raft_consensus.cc:3037] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:22.122056 29788 raft_consensus.cc:2454] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575473cf35164e2c9c383af36f3cea8d in term 1.
I20260925 02:26:22.122225 29656 raft_consensus.cc:2454] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575473cf35164e2c9c383af36f3cea8d in term 1.
I20260925 02:26:22.122229 29855 leader_election.cc:302] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575473cf35164e2c9c383af36f3cea8d, d7ba9837773544f89ba3d1081c030b9d; no voters:
I20260925 02:26:22.122319 29990 raft_consensus.cc:2781] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:22.122496 29990 raft_consensus.cc:686] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 1 LEADER]: Becoming Leader. State: Replica: 575473cf35164e2c9c383af36f3cea8d, State: Running, Role: LEADER
I20260925 02:26:22.122620 29990 consensus_queue.cc:237] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } }
I20260925 02:26:22.123425 29512 catalog_manager.cc:5495] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 575473cf35164e2c9c383af36f3cea8d (127.11.125.3). New cstate: current_term: 1 leader_uuid: "575473cf35164e2c9c383af36f3cea8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575473cf35164e2c9c383af36f3cea8d" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 37879 } health_report { overall_health: HEALTHY } } }
I20260925 02:26:22.154359 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.151839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.11.125.3:37879)
0925 02:26:22.151917 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:22.151919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:22.151945 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:26:22.152000 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:22.152002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:22.152021 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:22.152101 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:22.152240 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.152714 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.153543 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:22.153740 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:22.153812 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:22.153985 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:22.154095 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:22.154203 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:22.154222 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 02:26:22.154243 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:26:22.156412 29788 raft_consensus.cc:1261] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 1 FOLLOWER]: Refusing update from remote peer 575473cf35164e2c9c383af36f3cea8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:22.156424 29656 raft_consensus.cc:1261] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 1 FOLLOWER]: Refusing update from remote peer 575473cf35164e2c9c383af36f3cea8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:22.156816 29990 consensus_queue.cc:1035] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ba9837773544f89ba3d1081c030b9d" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:22.157038 29993 consensus_queue.cc:1035] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012d58c3806433b9b8127dc2140ba4a" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:22.159246 29996 mvcc.cc:205] Tried to move back new op lower bound from 7333081834109820928 to 7333081833974943744. Current Snapshot: MvccSnapshot[applied={T|T < 7333081834109820928}]
I20260925 02:26:22.159803 29997 mvcc.cc:205] Tried to move back new op lower bound from 7333081834109820928 to 7333081833974943744. Current Snapshot: MvccSnapshot[applied={T|T < 7333081834109820928}]
I20260925 02:26:22.160575 29998 mvcc.cc:205] Tried to move back new op lower bound from 7333081834109820928 to 7333081833974943744. Current Snapshot: MvccSnapshot[applied={T|T < 7333081834109820928 or (T in {7333081834109820928})}]
I20260925 02:26:22.162199 29512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:22.162297 29512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:22.163101 29512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 533c36a4bbc44b45887df050edfe4b4c: Sending DeleteTablet for 3 replicas of tablet 768f25c7bc3648048d5a54275d43d849
I20260925 02:26:22.163441 29767 tablet_service.cc:1514] Processing DeleteTablet for tablet 768f25c7bc3648048d5a54275d43d849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:22 UTC) from {username='slave'} at 127.0.0.1:45780
I20260925 02:26:22.163442 29635 tablet_service.cc:1514] Processing DeleteTablet for tablet 768f25c7bc3648048d5a54275d43d849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:22 UTC) from {username='slave'} at 127.0.0.1:43288
I20260925 02:26:22.163789 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet 768f25c7bc3648048d5a54275d43d849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:22 UTC) from {username='slave'} at 127.0.0.1:39444
I20260925 02:26:22.163836 30006 tablet_replica.cc:330] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a: stopping tablet replica
I20260925 02:26:22.163954 30006 raft_consensus.cc:2229] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:22.164072 30006 raft_consensus.cc:2258] T 768f25c7bc3648048d5a54275d43d849 P 8012d58c3806433b9b8127dc2140ba4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:22.164086 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29573
I20260925 02:26:22.164747 30007 tablet_replica.cc:330] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: stopping tablet replica
I20260925 02:26:22.164891 30007 raft_consensus.cc:2229] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:22.165040 30007 raft_consensus.cc:2258] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:22.167577 30007 ts_tablet_manager.cc:1905] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:22.169037 30008 tablet_replica.cc:330] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: stopping tablet replica
I20260925 02:26:22.169142 30008 raft_consensus.cc:2229] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:26:22.169248 30008 raft_consensus.cc:2258] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:22.169492 30007 ts_tablet_manager.cc:1918] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:22.169574 30007 log.cc:1196] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/768f25c7bc3648048d5a54275d43d849
I20260925 02:26:22.169705 30008 ts_tablet_manager.cc:1905] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:22.169860 30007 ts_tablet_manager.cc:1939] T 768f25c7bc3648048d5a54275d43d849 P d7ba9837773544f89ba3d1081c030b9d: Deleting consensus metadata
I20260925 02:26:22.170317 29499 catalog_manager.cc:4841] TS d7ba9837773544f89ba3d1081c030b9d (127.11.125.2:33691): tablet 768f25c7bc3648048d5a54275d43d849 (table test-table [id=fc1aec4118764b3699254a30602dba8e]) successfully deleted
I20260925 02:26:22.171294 30008 ts_tablet_manager.cc:1918] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:22.171402 30008 log.cc:1196] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/768f25c7bc3648048d5a54275d43d849
I20260925 02:26:22.171664 30008 ts_tablet_manager.cc:1939] T 768f25c7bc3648048d5a54275d43d849 P 575473cf35164e2c9c383af36f3cea8d: Deleting consensus metadata
I20260925 02:26:22.172163 29498 catalog_manager.cc:4841] TS 575473cf35164e2c9c383af36f3cea8d (127.11.125.3:37879): tablet 768f25c7bc3648048d5a54275d43d849 (table test-table [id=fc1aec4118764b3699254a30602dba8e]) successfully deleted
W20260925 02:26:22.173408 29496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:38811 (error 108)
I20260925 02:26:22.173954 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:22.173804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38811 (local address 127.0.0.1:43290)
0925 02:26:22.173867 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:22.173912 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:38811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:26:22.174005 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29705
W20260925 02:26:22.174849 29496 catalog_manager.cc:4584] TS 8012d58c3806433b9b8127dc2140ba4a (127.11.125.1:38811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 768f25c7bc3648048d5a54275d43d849: Network error: Client connection negotiation failed: client connection to 127.11.125.1:38811: connect: Connection refused (error 111)
I20260925 02:26:22.180768 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29837
I20260925 02:26:22.186321 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 29480
2026-09-25T02:26:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1005 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:22 dist-test-slave-hj7s krb5kdc[30012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:22 dist-test-slave-hj7s krb5kdc[30012](info): set up 2 sockets
Sep 25 02:26:22 dist-test-slave-hj7s krb5kdc[30012](info): commencing operation
krb5kdc: starting...
W20260925 02:26:23.230919 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:23.352228 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:41827
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:23.434216 30025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:23.434468 30025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:23.434507 30025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:23.435668 30025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:23.435740 30025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:23.435765 30025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:23.435787 30025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:23.435817 30025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:23.437353 30025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:23.437631 30025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:23.437863 30025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:23.442886 30031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:23.442956 30025 server_base.cc:1034] running on GCE node
W20260925 02:26:23.442970 30033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.442924 30030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:23.443616 30025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:23.444011 30025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:23.445186 30025 hybrid_clock.cc:648] HybridClock initialized: now 1790303183445173 us; error 28 us; skew 500 ppm
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:23.447659 30025 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:23.448139 30025 webserver.cc:460] Webserver started at http://127.11.125.62:40803/ using document root <none> and password file <none>
I20260925 02:26:23.448378 30025 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:23.448448 30025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:23.448611 30025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:23.449791 30025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "ea3e22243d19490d93d69cd44b3838f4"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "05b89d6dd902c0660967e116ec7c0752"
server_key_iv: "e1d8201aa62e29b4ca2fb31bbbf3b48a"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.450172 30025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "ea3e22243d19490d93d69cd44b3838f4"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "05b89d6dd902c0660967e116ec7c0752"
server_key_iv: "e1d8201aa62e29b4ca2fb31bbbf3b48a"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.451535 30025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:26:23.452284 30040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.452497 30025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:23.452606 30025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "ea3e22243d19490d93d69cd44b3838f4"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "05b89d6dd902c0660967e116ec7c0752"
server_key_iv: "e1d8201aa62e29b4ca2fb31bbbf3b48a"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.452687 30025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:23.460289 30025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:23.462183 30025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:23.462348 30025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:23.466444 30025 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:41827
I20260925 02:26:23.466475 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:41827 every 8 connection(s)
I20260925 02:26:23.466827 30025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:23.467288 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30025
I20260925 02:26:23.467422 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:23.467664 11764 external_mini_cluster.cc:1428] Setting key 2f92b747f328ea4c234dcb3cc6562d78
I20260925 02:26:23.467945 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:23.471472 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: Bootstrap starting.
I20260925 02:26:23.472172 30094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:23.472508 30094 log.cc:824] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:23.473215 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: No bootstrap required, opened a new log
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:23.475435 30094 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } }
I20260925 02:26:23.475597 30094 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:23.475623 30094 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3e22243d19490d93d69cd44b3838f4, State: Initialized, Role: FOLLOWER
I20260925 02:26:23.475775 30094 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } }
I20260925 02:26:23.475865 30094 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:23.475898 30094 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:23.475955 30094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:23.476720 30094 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } }
I20260925 02:26:23.476846 30094 leader_election.cc:302] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3e22243d19490d93d69cd44b3838f4; no voters:
I20260925 02:26:23.477035 30094 leader_election.cc:288] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:23.477068 30099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:23.477187 30099 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [term 1 LEADER]: Becoming Leader. State: Replica: ea3e22243d19490d93d69cd44b3838f4, State: Running, Role: LEADER
I20260925 02:26:23.477310 30094 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:23.477340 30099 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } }
I20260925 02:26:23.477674 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3e22243d19490d93d69cd44b3838f4. Latest consensus state: current_term: 1 leader_uuid: "ea3e22243d19490d93d69cd44b3838f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } } }
I20260925 02:26:23.477684 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3e22243d19490d93d69cd44b3838f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e22243d19490d93d69cd44b3838f4" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 41827 } } }
I20260925 02:26:23.477896 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:23.477882 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:23.478250 30108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:23.478863 30108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:23.479605 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:23.468285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.11.125.62:41827)
0925 02:26:23.468482 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:23.468492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:23.468527 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:26:23.469178 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:23.469186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:23.469236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:23.469377 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:23.470227 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.471101 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.472185 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.472691 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.472813 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:23.475496 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:23.475500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:23.475500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:23.475510 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:23.477602 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.478037 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.478039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.478040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.478140 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.478396 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.478397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.478397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.478717 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:23.478877 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:23.478987 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:26:23.479231 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260925 02:26:23.480507 30108 catalog_manager.cc:1347] Generated new cluster ID: a94c48d18f8e4d53aa0886e399c9dccb
I20260925 02:26:23.480563 30108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:23.518667 30108 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:23.519241 30108 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:23.522815 30108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: Generated new TSK 0
I20260925 02:26:23.523029 30108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:26:23.555608 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:23.663024 30120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:23.663383 30120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:23.663421 30120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:23.665300 30120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:23.665364 30120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:23.665409 30120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:23.666929 30120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:23.667186 30120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:23.667454 30120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:23.672688 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.672708 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.672780 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:23.673000 30120 server_base.cc:1034] running on GCE node
I20260925 02:26:23.673197 30120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:23.673445 30120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:23.674631 30120 hybrid_clock.cc:648] HybridClock initialized: now 1790303183674614 us; error 28 us; skew 500 ppm
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:23.677505 30120 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:26:23.677942 30120 webserver.cc:460] Webserver started at http://127.11.125.1:40611/ using document root <none> and password file <none>
I20260925 02:26:23.678261 30120 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:23.678333 30120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:23.678503 30120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:23.679625 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "8db05117fa05498684963e8209ce2194"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "159258ce930ec4f73f8cb32b692d3314"
server_key_iv: "929183560f16c7624dfce857ff58cec9"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.680006 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "8db05117fa05498684963e8209ce2194"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "159258ce930ec4f73f8cb32b692d3314"
server_key_iv: "929183560f16c7624dfce857ff58cec9"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.681555 30120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:23.682287 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.682482 30120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:23.682582 30120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "8db05117fa05498684963e8209ce2194"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "159258ce930ec4f73f8cb32b692d3314"
server_key_iv: "929183560f16c7624dfce857ff58cec9"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.682675 30120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:23.704030 30120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:23.705977 30120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:23.706140 30120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:23.706380 30120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:23.706692 30120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:23.706746 30120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.706790 30120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:23.706815 30120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.712769 30120 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:38755
I20260925 02:26:23.712817 30248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:38755 every 8 connection(s)
I20260925 02:26:23.713222 30120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:23.721711 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30120
I20260925 02:26:23.721830 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:23.722074 11764 external_mini_cluster.cc:1428] Setting key 3fb872e4b924eedd15a699014307193e
I20260925 02:26:23.722563 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:23.714021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:37151 (local address 127.11.125.62:41827)
0925 02:26:23.714085 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:23.714087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:23.714310 (+ 223us) server_negotiation.cc:407] Connection header received
0925 02:26:23.714854 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:23.714855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:23.714867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:23.714924 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:23.715902 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.716406 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.717430 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.717648 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.717762 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:23.719847 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:23.719849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:23.719849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:23.719856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:23.721125 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.721528 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.721529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.721530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.721588 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.721853 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.721854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.721855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.722014 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:23.722091 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:23.722384 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:26:23.722471 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:26:23.722905 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:23.713994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:41827 (local address 127.11.125.1:37151)
0925 02:26:23.714171 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:23.714214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:23.714734 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:23.714998 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:23.715002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:23.715589 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:23.715781 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:23.715786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.716504 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.716505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:23.717331 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:23.717333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.717763 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.717765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:23.717900 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:23.718425 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:23.718434 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:23.719747 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:23.721203 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:23.721211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:23.721213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:23.721448 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:23.721656 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:23.721657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:23.721658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:23.721794 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:23.722080 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:23.722088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:23.722244 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 02:26:23.722361 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 02:26:23.722518 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260925 02:26:23.723500 30249 heartbeater.cc:347] Connected to a master server at 127.11.125.62:41827
I20260925 02:26:23.723613 30249 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:23.723798 30249 heartbeater.cc:511] Master 127.11.125.62:41827 requested a full tablet report, sending...
I20260925 02:26:23.724295 30057 ts_manager.cc:194] Registered new tserver with Master: 8db05117fa05498684963e8209ce2194 (127.11.125.1:38755)
I20260925 02:26:23.725039 30057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:37151
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:26:23.747575 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:23.829442 30254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:23.829649 30254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:23.829674 30254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:23.830835 30254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:23.830878 30254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:23.830931 30254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:23.832374 30254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:23.832612 30254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:23.832849 30254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:23.837766 30259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.837869 30260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.837771 30262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:23.837867 30254 server_base.cc:1034] running on GCE node
I20260925 02:26:23.838169 30254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:23.838418 30254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:23.839588 30254 hybrid_clock.cc:648] HybridClock initialized: now 1790303183839582 us; error 36 us; skew 500 ppm
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:23.842011 30254 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:26:23.842427 30254 webserver.cc:460] Webserver started at http://127.11.125.2:38087/ using document root <none> and password file <none>
I20260925 02:26:23.842658 30254 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:23.842736 30254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:23.842895 30254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:23.843930 30254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "5b073e5edabe463ba308678806cbad62"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "ed7b0a70d7313726aacc8b4230ed99e0"
server_key_iv: "2fe892cb04c8b60de1f8bc49e7742906"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.844297 30254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "5b073e5edabe463ba308678806cbad62"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "ed7b0a70d7313726aacc8b4230ed99e0"
server_key_iv: "2fe892cb04c8b60de1f8bc49e7742906"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.845685 30254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:23.846390 30269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.846560 30254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:23.846658 30254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "5b073e5edabe463ba308678806cbad62"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "ed7b0a70d7313726aacc8b4230ed99e0"
server_key_iv: "2fe892cb04c8b60de1f8bc49e7742906"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.846740 30254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:23.853775 30254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:23.855515 30254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:23.855723 30254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:23.855954 30254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:23.856249 30254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:23.856303 30254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.856343 30254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:23.856371 30254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:23.862332 30254 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:45601
I20260925 02:26:23.862385 30384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:45601 every 8 connection(s)
I20260925 02:26:23.862684 30254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:23.871779 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:23.863378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:52203 (local address 127.11.125.62:41827)
0925 02:26:23.863448 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:23.863450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:23.863672 (+ 222us) server_negotiation.cc:407] Connection header received
0925 02:26:23.864179 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:23.864180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:23.864190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:23.864242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:23.865275 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.865845 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.866806 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.866997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.867078 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:23.869325 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:23.869326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:23.869326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:23.869333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:23.870565 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.870949 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.870950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.870951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.871008 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:23.871228 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:23.871229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:23.871229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:23.871349 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:23.871412 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:23.871644 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:26:23.871713 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:26:23.871948 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:23.863385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:41827 (local address 127.11.125.2:52203)
0925 02:26:23.863553 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:23.863589 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:23.864092 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:23.864309 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:23.864312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:23.864896 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:23.865174 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:23.865179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.865937 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.865937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:23.866715 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:23.866718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:23.867095 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:23.867096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:23.867204 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:23.867685 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:23.867693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:23.869187 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:23.870637 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:23.870644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:23.870646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:23.870866 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:23.871073 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:23.871074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:23.871074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:23.871171 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:23.871418 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:23.871423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:23.871534 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 02:26:23.871616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:26:23.871733 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260925 02:26:23.872112 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30254
I20260925 02:26:23.872236 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:23.872442 11764 external_mini_cluster.cc:1428] Setting key c751205afd1b1d0c80e6a1681ac7b3ca
I20260925 02:26:23.872431 30385 heartbeater.cc:347] Connected to a master server at 127.11.125.62:41827
I20260925 02:26:23.872550 30385 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:23.872785 30385 heartbeater.cc:511] Master 127.11.125.62:41827 requested a full tablet report, sending...
I20260925 02:26:23.873252 30057 ts_manager.cc:194] Registered new tserver with Master: 5b073e5edabe463ba308678806cbad62 (127.11.125.2:45601)
I20260925 02:26:23.873634 30057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:52203
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:26:23.899757 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:23.983322 30390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:23.983518 30390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:23.983541 30390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:23.984599 30390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:23.984632 30390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:23.984658 30390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:23.986232 30390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:23.986420 30390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:23.986589 30390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:26:23.991672 30390 server_base.cc:1034] running on GCE node
W20260925 02:26:23.991643 30398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.991634 30395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:23.991641 30396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:23.992090 30390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:23.992285 30390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:23.993422 30390 hybrid_clock.cc:648] HybridClock initialized: now 1790303183993407 us; error 29 us; skew 500 ppm
Sep 25 02:26:23 dist-test-slave-hj7s krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:23.995865 30390 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:26:23.996353 30390 webserver.cc:460] Webserver started at http://127.11.125.3:39561/ using document root <none> and password file <none>
I20260925 02:26:23.996594 30390 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:23.996639 30390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:23.996803 30390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:23.997853 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "26d157a6274c43968d94b457f7e790ce"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "e972ceb93e5ea6bc8ba05b383d95bea7"
server_key_iv: "5b93ae4b533e28c299a3f3ac565b613f"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.998204 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "26d157a6274c43968d94b457f7e790ce"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "e972ceb93e5ea6bc8ba05b383d95bea7"
server_key_iv: "5b93ae4b533e28c299a3f3ac565b613f"
server_key_version: "encryptionkey@0"
I20260925 02:26:23.999571 30390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:26:24.000279 30405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.000471 30390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:24.000574 30390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "26d157a6274c43968d94b457f7e790ce"
format_stamp: "Formatted at 2026-09-25 02:26:23 on dist-test-slave-hj7s"
server_key: "e972ceb93e5ea6bc8ba05b383d95bea7"
server_key_iv: "5b93ae4b533e28c299a3f3ac565b613f"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.000658 30390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:24.028935 30390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:24.030872 30390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:24.031036 30390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:24.031272 30390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:24.031611 30390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:24.031667 30390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.031711 30390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:24.031739 30390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.037751 30390 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:45633
I20260925 02:26:24.037801 30519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:45633 every 8 connection(s)
I20260925 02:26:24.038113 30390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:26:24 dist-test-slave-hj7s krb5kdc[30012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:24.046808 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30390
I20260925 02:26:24.046914 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:24.047132 11764 external_mini_cluster.cc:1428] Setting key c358e49314748c96a18a711217bf948d
I20260925 02:26:24.047338 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.038764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:49621 (local address 127.11.125.62:41827)
0925 02:26:24.038808 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.038810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.039104 (+ 294us) server_negotiation.cc:407] Connection header received
0925 02:26:24.039629 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.039630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.039643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.039707 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:24.040654 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.041217 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.042249 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.042452 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.042529 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.044590 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:24.044591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:24.044592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:24.044599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:24.045848 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:24.046238 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:24.046239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:24.046239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:24.046299 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:24.046571 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:24.046572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:24.046573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:24.046731 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:24.046807 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.047143 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.047240 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:26:24.047533 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.038751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:41827 (local address 127.11.125.3:49621)
0925 02:26:24.038939 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.038997 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.039520 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.039792 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.039796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.040360 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:24.040550 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.040554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.041333 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.041334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.042144 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.042147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.042536 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.042538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.042669 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.043098 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:24.043106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:24.044494 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:24.045931 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:24.045939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:24.045942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:24.046160 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:24.046381 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:24.046382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:24.046382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:24.046495 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:24.046809 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:24.046816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:24.046986 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.047098 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.047268 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260925 02:26:24.048036 30520 heartbeater.cc:347] Connected to a master server at 127.11.125.62:41827
I20260925 02:26:24.048139 30520 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:24.048321 30520 heartbeater.cc:511] Master 127.11.125.62:41827 requested a full tablet report, sending...
I20260925 02:26:24.048731 30057 ts_manager.cc:194] Registered new tserver with Master: 26d157a6274c43968d94b457f7e790ce (127.11.125.3:45633)
I20260925 02:26:24.049167 30057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:49621
I20260925 02:26:24.059047 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:24.066428 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.060319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.11.125.62:41827)
0925 02:26:24.060367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.060369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.060515 (+ 146us) server_negotiation.cc:407] Connection header received
0925 02:26:24.060555 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.060556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.060565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.060612 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:24.061309 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.061835 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.062557 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.062741 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.062799 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.064073 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:24.064074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:24.064074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:24.064078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:24.065311 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:24.065720 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:24.065721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:24.065721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:24.065770 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:24.065964 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:24.065965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:24.065965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:24.066080 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:24.066158 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.066255 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.066301 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:26:24.068233 30057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:24.069978 30057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:24.076632 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.11.125.3:45633)
0925 02:26:24.072878 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.072884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.072966 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:26:24.073047 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.073049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.073065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.073233 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:24.073486 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074496 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.075766 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076252 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.076512 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.076554 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260925 02:26:24.076625 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:45633 (local address 127.0.0.1:46968)
0925 02:26:24.072883 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.072893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.072980 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.073315 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.073316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.073318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:24.073357 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.073359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074608 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.074609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.075680 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.075681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076365 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.076428 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076433 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.076490 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.076540 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":59,"threads_started":1}
I20260925 02:26:24.076707 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.11.125.2:45601)
0925 02:26:24.072629 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.072635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.072646 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:24.072689 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.072692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.072709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.072804 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:24.073028 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074196 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.075321 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076050 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076122 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076180 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.076587 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.076627 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260925 02:26:24.076541 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:45601 (local address 127.0.0.1:58962)
0925 02:26:24.072456 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.072498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.072570 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.072826 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.072829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.072831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:24.072933 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.072935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074308 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.074309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.075234 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.075236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076217 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.076303 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076318 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.076396 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.076454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":49,"threads_started":1}
I20260925 02:26:24.076812 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.11.125.1:38755)
0925 02:26:24.073007 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.073012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.073022 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:24.073068 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.073070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.073086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.073379 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:24.073461 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074114 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.075572 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076314 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076588 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076654 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.076678 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.076722 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":112,"threads_started":1}
I20260925 02:26:24.076943 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.072264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38755 (local address 127.0.0.1:58034)
0925 02:26:24.072561 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.072571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.072612 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.073204 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.073206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.073208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:24.073262 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.073264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.074226 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.074227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.075481 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.075484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.076413 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.076414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.076467 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.076471 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.076499 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.076531 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260925 02:26:24.077641 30317 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7553936b264e02946e83e9d9869d22 (DEFAULT_TABLE table=test-table [id=489242de96db449aac39b0ce314f1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:24.077782 30183 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7553936b264e02946e83e9d9869d22 (DEFAULT_TABLE table=test-table [id=489242de96db449aac39b0ce314f1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:24.078102 30317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7553936b264e02946e83e9d9869d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:24.078189 30183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7553936b264e02946e83e9d9869d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:24.078151 30454 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7553936b264e02946e83e9d9869d22 (DEFAULT_TABLE table=test-table [id=489242de96db449aac39b0ce314f1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:24.078464 30454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7553936b264e02946e83e9d9869d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:24.080793 30538 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Bootstrap starting.
I20260925 02:26:24.081583 30538 tablet_bootstrap.cc:654] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:24.081573 30539 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Bootstrap starting.
I20260925 02:26:24.082283 30539 tablet_bootstrap.cc:654] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:24.082283 30538 log.cc:824] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:24.082872 30539 log.cc:824] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:24.083101 30538 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: No bootstrap required, opened a new log
I20260925 02:26:24.083179 30538 ts_tablet_manager.cc:1397] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:24.083225 30540 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Bootstrap starting.
I20260925 02:26:24.083578 30539 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: No bootstrap required, opened a new log
I20260925 02:26:24.083663 30539 ts_tablet_manager.cc:1397] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:24.083870 30540 tablet_bootstrap.cc:654] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:24.084613 30540 log.cc:824] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:24.085526 30540 tablet_bootstrap.cc:492] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: No bootstrap required, opened a new log
I20260925 02:26:24.085594 30540 ts_tablet_manager.cc:1397] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:24.085503 30538 raft_consensus.cc:348] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.085700 30538 raft_consensus.cc:374] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:24.085754 30538 raft_consensus.cc:729] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db05117fa05498684963e8209ce2194, State: Initialized, Role: FOLLOWER
I20260925 02:26:24.085660 30539 raft_consensus.cc:348] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.085882 30539 raft_consensus.cc:374] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:24.085930 30539 raft_consensus.cc:729] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b073e5edabe463ba308678806cbad62, State: Initialized, Role: FOLLOWER
I20260925 02:26:24.085922 30538 consensus_queue.cc:260] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.086086 30539 consensus_queue.cc:260] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.086277 30249 heartbeater.cc:503] Master 127.11.125.62:41827 was elected leader, sending a full tablet report...
I20260925 02:26:24.086377 30539 ts_tablet_manager.cc:1428] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:24.086215 30538 ts_tablet_manager.cc:1428] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:24.086519 30385 heartbeater.cc:503] Master 127.11.125.62:41827 was elected leader, sending a full tablet report...
I20260925 02:26:24.088006 30540 raft_consensus.cc:348] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.088131 30540 raft_consensus.cc:374] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:24.088152 30540 raft_consensus.cc:729] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d157a6274c43968d94b457f7e790ce, State: Initialized, Role: FOLLOWER
I20260925 02:26:24.088274 30540 consensus_queue.cc:260] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.088541 30540 ts_tablet_manager.cc:1428] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:24.088581 30520 heartbeater.cc:503] Master 127.11.125.62:41827 was elected leader, sending a full tablet report...
W20260925 02:26:24.113207 30386 tablet.cc:2369] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:24.214022 30250 tablet.cc:2369] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:24.288672 30522 tablet.cc:2369] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:24.301223 30546 raft_consensus.cc:482] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:24.301371 30546 raft_consensus.cc:504] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.302238 30546 leader_election.cc:288] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db05117fa05498684963e8209ce2194 (127.11.125.1:38755), 5b073e5edabe463ba308678806cbad62 (127.11.125.2:45601)
I20260925 02:26:24.304880 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.302145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:60297 (local address 127.11.125.2:45601)
0925 02:26:24.302193 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.302195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.302233 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:26:24.302267 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.302269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.302279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.302331 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:24.302505 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.303001 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304082 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.304515 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.304612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.304770 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.304804 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 02:26:24.304888 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.302080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:45601 (local address 127.11.125.3:60297)
0925 02:26:24.302136 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.302149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.302189 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.302367 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.302368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.302370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:24.302442 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.302444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.303103 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.303105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.303985 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.303988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.304601 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.304684 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.304687 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.304762 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.304803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:26:24.305064 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.302308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:35913 (local address 127.11.125.1:38755)
0925 02:26:24.302361 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.302363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.302372 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:24.302398 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.302399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.302409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.302466 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:24.302645 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.303109 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304092 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.304643 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304720 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.304728 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.304895 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.304969 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:26:24.305244 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.302098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38755 (local address 127.11.125.3:35913)
0925 02:26:24.302286 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.302296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.302339 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.302492 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.302493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.302495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:24.302571 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.302572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.303202 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.303204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.303992 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.303995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.304735 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.304736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.304815 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.304818 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.304888 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.305102 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260925 02:26:24.305255 30338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7553936b264e02946e83e9d9869d22" candidate_uuid: "26d157a6274c43968d94b457f7e790ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b073e5edabe463ba308678806cbad62" is_pre_election: true
I20260925 02:26:24.305477 30338 raft_consensus.cc:2454] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d157a6274c43968d94b457f7e790ce in term 0.
I20260925 02:26:24.305645 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7553936b264e02946e83e9d9869d22" candidate_uuid: "26d157a6274c43968d94b457f7e790ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db05117fa05498684963e8209ce2194" is_pre_election: true
I20260925 02:26:24.305713 30409 leader_election.cc:302] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d157a6274c43968d94b457f7e790ce, 5b073e5edabe463ba308678806cbad62; no voters:
I20260925 02:26:24.305814 30203 raft_consensus.cc:2454] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d157a6274c43968d94b457f7e790ce in term 0.
I20260925 02:26:24.305853 30546 raft_consensus.cc:2781] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:24.305932 30546 raft_consensus.cc:482] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:24.305958 30546 raft_consensus.cc:3037] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:24.306793 30546 raft_consensus.cc:504] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.306985 30546 leader_election.cc:288] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [CANDIDATE]: Term 1 election: Requested vote from peers 8db05117fa05498684963e8209ce2194 (127.11.125.1:38755), 5b073e5edabe463ba308678806cbad62 (127.11.125.2:45601)
I20260925 02:26:24.307156 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7553936b264e02946e83e9d9869d22" candidate_uuid: "26d157a6274c43968d94b457f7e790ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db05117fa05498684963e8209ce2194"
I20260925 02:26:24.307235 30203 raft_consensus.cc:3037] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:24.307329 30338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7553936b264e02946e83e9d9869d22" candidate_uuid: "26d157a6274c43968d94b457f7e790ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b073e5edabe463ba308678806cbad62"
I20260925 02:26:24.307413 30338 raft_consensus.cc:3037] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:24.308203 30203 raft_consensus.cc:2454] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d157a6274c43968d94b457f7e790ce in term 1.
I20260925 02:26:24.308360 30338 raft_consensus.cc:2454] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d157a6274c43968d94b457f7e790ce in term 1.
I20260925 02:26:24.308421 30406 leader_election.cc:302] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d157a6274c43968d94b457f7e790ce, 8db05117fa05498684963e8209ce2194; no voters:
I20260925 02:26:24.308539 30546 raft_consensus.cc:2781] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:24.308745 30546 raft_consensus.cc:686] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 1 LEADER]: Becoming Leader. State: Replica: 26d157a6274c43968d94b457f7e790ce, State: Running, Role: LEADER
I20260925 02:26:24.308867 30546 consensus_queue.cc:237] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } }
I20260925 02:26:24.309780 30057 catalog_manager.cc:5495] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d157a6274c43968d94b457f7e790ce (127.11.125.3). New cstate: current_term: 1 leader_uuid: "26d157a6274c43968d94b457f7e790ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d157a6274c43968d94b457f7e790ce" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260925 02:26:24.363284 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.360525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.11.125.3:45633)
0925 02:26:24.360593 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.360595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.360605 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:24.360672 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.360673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.360702 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.360779 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:24.361007 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.361551 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.362423 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.362648 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.362731 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.362889 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:24.362992 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:24.363126 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.363170 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.363181 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:26:24.365307 30338 raft_consensus.cc:1261] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 1 FOLLOWER]: Refusing update from remote peer 26d157a6274c43968d94b457f7e790ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:24.365307 30203 raft_consensus.cc:1261] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 1 FOLLOWER]: Refusing update from remote peer 26d157a6274c43968d94b457f7e790ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:24.365684 30548 consensus_queue.cc:1035] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db05117fa05498684963e8209ce2194" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 38755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:24.365839 30546 consensus_queue.cc:1035] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b073e5edabe463ba308678806cbad62" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:24.368386 30551 mvcc.cc:205] Tried to move back new op lower bound from 7333081843157442560 to 7333081842929836032. Current Snapshot: MvccSnapshot[applied={T|T < 7333081843157442560}]
I20260925 02:26:24.371213 30057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:24.371336 30057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:24.372298 30057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e22243d19490d93d69cd44b3838f4: Sending DeleteTablet for 3 replicas of tablet 0e7553936b264e02946e83e9d9869d22
I20260925 02:26:24.372649 30183 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7553936b264e02946e83e9d9869d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:24 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:58034
I20260925 02:26:24.372748 30317 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7553936b264e02946e83e9d9869d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:24 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:58962
I20260925 02:26:24.372890 30454 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7553936b264e02946e83e9d9869d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:24 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46968
I20260925 02:26:24.373059 30561 tablet_replica.cc:330] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194: stopping tablet replica
I20260925 02:26:24.373184 30561 raft_consensus.cc:2229] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:24.373194 30563 tablet_replica.cc:330] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: stopping tablet replica
I20260925 02:26:24.373324 30561 raft_consensus.cc:2258] T 0e7553936b264e02946e83e9d9869d22 P 8db05117fa05498684963e8209ce2194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:24.373342 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30120
I20260925 02:26:24.375355 30562 tablet_replica.cc:330] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: stopping tablet replica
I20260925 02:26:24.375494 30562 raft_consensus.cc:2229] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:24.375648 30562 raft_consensus.cc:2258] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:24.376075 30562 ts_tablet_manager.cc:1905] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:24.376992 30563 raft_consensus.cc:2229] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:26:24.377177 30563 raft_consensus.cc:2258] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:24.377697 30563 ts_tablet_manager.cc:1905] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:24.377923 30562 ts_tablet_manager.cc:1918] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:24.378010 30562 log.cc:1196] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/0e7553936b264e02946e83e9d9869d22
I20260925 02:26:24.378389 30562 ts_tablet_manager.cc:1939] T 0e7553936b264e02946e83e9d9869d22 P 5b073e5edabe463ba308678806cbad62: Deleting consensus metadata
I20260925 02:26:24.379848 30044 catalog_manager.cc:4841] TS 5b073e5edabe463ba308678806cbad62 (127.11.125.2:45601): tablet 0e7553936b264e02946e83e9d9869d22 (table test-table [id=489242de96db449aac39b0ce314f1971]) successfully deleted
I20260925 02:26:24.380164 30563 ts_tablet_manager.cc:1918] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:24.380260 30563 log.cc:1196] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/0e7553936b264e02946e83e9d9869d22
I20260925 02:26:24.380538 30563 ts_tablet_manager.cc:1939] T 0e7553936b264e02946e83e9d9869d22 P 26d157a6274c43968d94b457f7e790ce: Deleting consensus metadata
I20260925 02:26:24.381064 30044 catalog_manager.cc:4841] TS 26d157a6274c43968d94b457f7e790ce (127.11.125.3:45633): tablet 0e7553936b264e02946e83e9d9869d22 (table test-table [id=489242de96db449aac39b0ce314f1971]) successfully deleted
W20260925 02:26:24.381850 30041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:38755 (error 108)
I20260925 02:26:24.381906 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30254
I20260925 02:26:24.383502 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.383371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:38755 (local address 127.0.0.1:58038)
0925 02:26:24.383422 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.383458 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:38755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:26:24.383647 30041 catalog_manager.cc:4584] TS 8db05117fa05498684963e8209ce2194 (127.11.125.1:38755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e7553936b264e02946e83e9d9869d22: Network error: Client connection negotiation failed: client connection to 127.11.125.1:38755: connect: Connection refused (error 111)
I20260925 02:26:24.388377 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30390
I20260925 02:26:24.392989 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30025
2026-09-25T02:26:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:24Z Disabled control of system clock
I20260925 02:26:24.420620 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39631
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:24.502871 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:24.503078 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:24.503101 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:24.504302 30568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:24.504344 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:24.504357 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:24.504370 30568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:24.504420 30568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:24.505760 30568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:39631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:24.505992 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:24.506285 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:24.511101 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.511157 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:24.511368 30568 server_base.cc:1034] running on GCE node
W20260925 02:26:24.511137 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:24.511698 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:24.511982 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:24.513147 30568 hybrid_clock.cc:648] HybridClock initialized: now 1790303184513131 us; error 26 us; skew 500 ppm
I20260925 02:26:24.521095 30568 webserver.cc:460] Webserver started at http://127.11.125.62:33119/ using document root <none> and password file <none>
I20260925 02:26:24.521370 30568 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:24.521445 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:24.521600 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:24.522727 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "9efd37d4f52348ea9fc67288a91ffb49"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "655aeab2cbeea5dbb61f218586fdd33d"
server_key_iv: "5ca4ef1c554d32717e0bdbd28fd49b60"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.523090 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "9efd37d4f52348ea9fc67288a91ffb49"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "655aeab2cbeea5dbb61f218586fdd33d"
server_key_iv: "5ca4ef1c554d32717e0bdbd28fd49b60"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.524420 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:24.525259 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.525447 30568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.525549 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "9efd37d4f52348ea9fc67288a91ffb49"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "655aeab2cbeea5dbb61f218586fdd33d"
server_key_iv: "5ca4ef1c554d32717e0bdbd28fd49b60"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.525636 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:24.540408 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:24.540704 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:24.540863 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:24.544800 30568 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:39631
I20260925 02:26:24.544847 30634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:39631 every 8 connection(s)
I20260925 02:26:24.545238 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:24.546227 30635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:24.548521 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: Bootstrap starting.
I20260925 02:26:24.549180 30635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:24.549489 30635 log.cc:824] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:24.550185 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: No bootstrap required, opened a new log
I20260925 02:26:24.551800 30635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } }
I20260925 02:26:24.551959 30635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:24.552009 30635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efd37d4f52348ea9fc67288a91ffb49, State: Initialized, Role: FOLLOWER
I20260925 02:26:24.552160 30635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } }
I20260925 02:26:24.552222 30635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:24.552244 30635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:24.552315 30635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:24.552841 30635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } }
I20260925 02:26:24.552992 30635 leader_election.cc:302] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9efd37d4f52348ea9fc67288a91ffb49; no voters:
I20260925 02:26:24.553128 30635 leader_election.cc:288] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:24.553184 30639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:24.553345 30639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [term 1 LEADER]: Becoming Leader. State: Replica: 9efd37d4f52348ea9fc67288a91ffb49, State: Running, Role: LEADER
I20260925 02:26:24.553404 30635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:24.553447 30639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } }
I20260925 02:26:24.553889 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9efd37d4f52348ea9fc67288a91ffb49. Latest consensus state: current_term: 1 leader_uuid: "9efd37d4f52348ea9fc67288a91ffb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } } }
I20260925 02:26:24.553978 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:24.553966 30640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9efd37d4f52348ea9fc67288a91ffb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efd37d4f52348ea9fc67288a91ffb49" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39631 } } }
I20260925 02:26:24.554081 30640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:24.554360 30648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:24.554876 30648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:24.555213 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30568
I20260925 02:26:24.555334 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:24.555538 11764 external_mini_cluster.cc:1428] Setting key 4f70c098e1c48ff19c350bafacd7f917
I20260925 02:26:24.557163 30648 catalog_manager.cc:1347] Generated new cluster ID: 4413d89d8cd145a7ae27d39038ececd5
I20260925 02:26:24.557219 30648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:24.560778 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.556179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.11.125.62:39631)
0925 02:26:24.556374 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.556389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.556413 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:26:24.556505 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.556510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.556597 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.556735 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:24.556996 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.558036 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.558826 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.559251 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.559408 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.560091 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:24.560094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:24.560095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:24.560277 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:24.560311 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:24.560322 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:24.560402 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.560478 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.560560 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260925 02:26:24.565040 30648 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:24.565691 30648 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:24.569686 30648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: Generated new TSK 0
I20260925 02:26:24.569880 30648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:24.577183 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--builtin_ntp_servers=127.11.125.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:24.685410 30660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:24.685631 30660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:24.685667 30660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:24.686822 30660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:24.686872 30660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:24.686906 30660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:24.688645 30660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:24.688843 30660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:24.689152 30660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:24.694132 30665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.694317 30666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.694424 30668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:24.694868 30660 server_base.cc:1034] running on GCE node
I20260925 02:26:24.695072 30660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:24.695353 30660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:24.696516 30660 hybrid_clock.cc:648] HybridClock initialized: now 1790303184696504 us; error 27 us; skew 500 ppm
I20260925 02:26:24.698012 30660 webserver.cc:460] Webserver started at http://127.11.125.1:43031/ using document root <none> and password file <none>
I20260925 02:26:24.698292 30660 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:24.698341 30660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:24.698477 30660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:24.699546 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "d87e828eeaf040529ad6829f07272a5d"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "078129c44f2b35e2aebff9274811070d"
server_key_iv: "da3d050a5f59534f4ac96e04f5dac2bd"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.699913 30660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "d87e828eeaf040529ad6829f07272a5d"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "078129c44f2b35e2aebff9274811070d"
server_key_iv: "da3d050a5f59534f4ac96e04f5dac2bd"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.701329 30660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 02:26:24.702140 30674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.702338 30660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:24.702406 30660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "d87e828eeaf040529ad6829f07272a5d"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "078129c44f2b35e2aebff9274811070d"
server_key_iv: "da3d050a5f59534f4ac96e04f5dac2bd"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.702494 30660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:24.732731 30660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:24.733183 30660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:24.733361 30660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:24.733686 30660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:24.734099 30660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:24.734134 30660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.734192 30660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:24.734228 30660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.740381 30660 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:42959
I20260925 02:26:24.740437 30787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:42959 every 8 connection(s)
I20260925 02:26:24.740784 30660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:24.744732 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30660
I20260925 02:26:24.744853 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:24.745075 11764 external_mini_cluster.cc:1428] Setting key 2dab03ee65011fc88495d30d623b2d27
I20260925 02:26:24.746165 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--builtin_ntp_servers=127.11.125.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:26:24.746625 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.741591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:38197 (local address 127.11.125.62:39631)
0925 02:26:24.741668 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.741670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.741958 (+ 288us) server_negotiation.cc:407] Connection header received
0925 02:26:24.742473 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.742474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.742488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.742544 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:24.743247 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.743743 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.744933 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.745167 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.745237 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.746174 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:24.746176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:24.746177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:24.746184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:24.746202 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:24.746207 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:24.746275 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.746462 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.746495 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:26:24.746801 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.741595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39631 (local address 127.11.125.1:38197)
0925 02:26:24.741820 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.741858 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.742382 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.742624 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.742627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.742640 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:24.743152 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.743156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.743825 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.743826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.744783 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.744787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.745290 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.745292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.745426 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.746027 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:24.746027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:24.746057 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:24.746081 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:24.746302 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:24.746309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:24.746323 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.746460 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.746566 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260925 02:26:24.747316 30788 heartbeater.cc:347] Connected to a master server at 127.11.125.62:39631
I20260925 02:26:24.747429 30788 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:24.747627 30788 heartbeater.cc:511] Master 127.11.125.62:39631 requested a full tablet report, sending...
I20260925 02:26:24.748181 30599 ts_manager.cc:194] Registered new tserver with Master: d87e828eeaf040529ad6829f07272a5d (127.11.125.1:42959)
I20260925 02:26:24.748821 30599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.1:38197
W20260925 02:26:24.827697 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:24.827888 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:24.827909 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:24.829135 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:24.829176 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:24.829201 30791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:24.830544 30791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:24.830794 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:24.831169 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:24.836127 30797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.836164 30799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.836175 30796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:24.836294 30791 server_base.cc:1034] running on GCE node
I20260925 02:26:24.836588 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:24.836822 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:24.837982 30791 hybrid_clock.cc:648] HybridClock initialized: now 1790303184837991 us; error 48 us; skew 500 ppm
I20260925 02:26:24.839341 30791 webserver.cc:460] Webserver started at http://127.11.125.2:45363/ using document root <none> and password file <none>
I20260925 02:26:24.839586 30791 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:24.839656 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:24.839855 30791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:24.841015 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "442f0c25ca534010b1290a492213884e"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "d9d3c4d8a4b3ee8bb93b091f3cec92cb"
server_key_iv: "72982432fbae59d6aeb9b3f3831c80d1"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.841385 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "442f0c25ca534010b1290a492213884e"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "d9d3c4d8a4b3ee8bb93b091f3cec92cb"
server_key_iv: "72982432fbae59d6aeb9b3f3831c80d1"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.842684 30791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:24.843395 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.843564 30791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:24.843699 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "442f0c25ca534010b1290a492213884e"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "d9d3c4d8a4b3ee8bb93b091f3cec92cb"
server_key_iv: "72982432fbae59d6aeb9b3f3831c80d1"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.843794 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:24.864007 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:24.864331 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:24.864500 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:24.864745 30791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:24.865154 30791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:24.865219 30791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.865263 30791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:24.865290 30791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.871518 30791 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:37817
I20260925 02:26:24.871582 30919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:37817 every 8 connection(s)
I20260925 02:26:24.871902 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:26:24.877849 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.872628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:46147 (local address 127.11.125.62:39631)
0925 02:26:24.872698 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.872700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.872956 (+ 256us) server_negotiation.cc:407] Connection header received
0925 02:26:24.873669 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:24.873671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:24.873685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:24.873740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:24.874525 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.875167 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.876287 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.876545 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.876642 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.877503 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:24.877505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:24.877505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:24.877510 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:24.877526 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:24.877530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:24.877603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:24.877719 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 02:26:24.877749 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:26:24.878122 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.872598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39631 (local address 127.11.125.2:46147)
0925 02:26:24.872816 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.872850 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:24.873439 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:24.873818 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:24.873822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:24.873833 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:24.874408 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.874414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.875302 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.875303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.876170 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:24.876173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:24.876649 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:24.876651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:24.876802 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:24.877341 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:24.877342 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:24.877372 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:24.877397 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:24.877618 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:24.877625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:24.877628 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:24.877705 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:26:24.877815 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260925 02:26:24.878726 30920 heartbeater.cc:347] Connected to a master server at 127.11.125.62:39631
I20260925 02:26:24.878837 30920 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:24.879072 30920 heartbeater.cc:511] Master 127.11.125.62:39631 requested a full tablet report, sending...
I20260925 02:26:24.879487 30599 ts_manager.cc:194] Registered new tserver with Master: 442f0c25ca534010b1290a492213884e (127.11.125.2:37817)
I20260925 02:26:24.879832 30599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.2:46147
I20260925 02:26:24.881721 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30791
I20260925 02:26:24.881814 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:24.881968 11764 external_mini_cluster.cc:1428] Setting key f3f9eef28e99c4a19311233516c6b8e1
I20260925 02:26:24.882825 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--builtin_ntp_servers=127.11.125.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:24.963762 30923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:24.963970 30923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:24.963992 30923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:24.965099 30923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:24.965132 30923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:24.965157 30923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:24.966446 30923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:24.966626 30923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:24.966879 30923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:24.971938 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.971939 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:24.971987 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:24.972045 30923 server_base.cc:1034] running on GCE node
I20260925 02:26:24.972399 30923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:24.972649 30923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:24.973825 30923 hybrid_clock.cc:648] HybridClock initialized: now 1790303184973810 us; error 50 us; skew 500 ppm
I20260925 02:26:24.975199 30923 webserver.cc:460] Webserver started at http://127.11.125.3:34383/ using document root <none> and password file <none>
I20260925 02:26:24.975474 30923 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:24.975546 30923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:24.975703 30923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:24.976791 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "a67e54a454b44930a59a45feed87dd5b"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "c030199452aa245e2618ce21fdbe0526"
server_key_iv: "d2f70482a2c09bd430ed75531d5da84c"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.977242 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "a67e54a454b44930a59a45feed87dd5b"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "c030199452aa245e2618ce21fdbe0526"
server_key_iv: "d2f70482a2c09bd430ed75531d5da84c"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.978580 30923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:24.979422 30937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.979597 30923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:24.979689 30923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "a67e54a454b44930a59a45feed87dd5b"
format_stamp: "Formatted at 2026-09-25 02:26:24 on dist-test-slave-hj7s"
server_key: "c030199452aa245e2618ce21fdbe0526"
server_key_iv: "d2f70482a2c09bd430ed75531d5da84c"
server_key_version: "encryptionkey@0"
I20260925 02:26:24.979781 30923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:24.991333 30923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:24.991636 30923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:24.991787 30923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:24.992027 30923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:24.992347 30923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:24.992403 30923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.992443 30923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:24.992472 30923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:24.998312 30923 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:34627
I20260925 02:26:24.998371 31051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:34627 every 8 connection(s)
I20260925 02:26:24.998672 30923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:26:25.003770 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.999387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:45697 (local address 127.11.125.62:39631)
0925 02:26:24.999449 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:24.999452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:24.999690 (+ 238us) server_negotiation.cc:407] Connection header received
0925 02:26:25.000148 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.000149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.000159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.000212 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:25.000823 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.001331 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.002366 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.002540 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.002590 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.003571 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:25.003572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:25.003572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:25.003576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:25.003586 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:25.003589 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:25.003627 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.003693 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.003713 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:26:25.004016 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:24.999386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39631 (local address 127.11.125.3:45697)
0925 02:26:24.999576 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:24.999609 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.000068 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.000261 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.000264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.000273 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:25.000759 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.000763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.001459 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.001461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.002280 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.002282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.002682 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.002683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.002805 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.003441 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:25.003441 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:25.003470 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:25.003495 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:25.003629 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:25.003632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:25.003634 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.003685 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.003761 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260925 02:26:25.004635 31052 heartbeater.cc:347] Connected to a master server at 127.11.125.62:39631
I20260925 02:26:25.004748 31052 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:25.005020 31052 heartbeater.cc:511] Master 127.11.125.62:39631 requested a full tablet report, sending...
I20260925 02:26:25.005414 30599 ts_manager.cc:194] Registered new tserver with Master: a67e54a454b44930a59a45feed87dd5b (127.11.125.3:34627)
I20260925 02:26:25.005788 30599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.3:45697
I20260925 02:26:25.007462 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 30923
I20260925 02:26:25.007550 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:25.007692 11764 external_mini_cluster.cc:1428] Setting key ea1a33be78800e740c32e40bd7942f0c
I20260925 02:26:25.008615 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:25.012560 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.009957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.11.125.62:39631)
0925 02:26:25.009998 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.010000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.010121 (+ 121us) server_negotiation.cc:407] Connection header received
0925 02:26:25.010153 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.010153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.010164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.010200 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:25.010331 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.010764 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.011438 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.011606 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.011654 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.012305 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:25.012306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:25.012306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:25.012310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:25.012319 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:25.012323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:25.012383 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.012445 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.012453 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:25.014273 30599 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:25.015733 30599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:25.021806 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.017863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42959 (local address 127.0.0.1:56826)
0925 02:26:25.018038 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.018079 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.018142 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.018440 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.018443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.018446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:25.018510 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.018512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.019540 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.019541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.020545 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.020548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.021515 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.021517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.021591 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.021603 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.021678 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.021731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":49,"threads_started":1}
I20260925 02:26:25.021976 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.017874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:37817 (local address 127.0.0.1:39428)
0925 02:26:25.018171 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.018181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.018228 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.018518 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.018520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.018522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:25.018569 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.018571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.019858 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.019860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.020799 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.020802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.021748 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.021749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.021807 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.021811 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.021860 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.021901 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":95,"threads_started":1}
I20260925 02:26:25.021816 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.017958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.11.125.1:42959)
0925 02:26:25.018271 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.018275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.018286 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:25.018329 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.018331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.018347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.018436 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:25.018599 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.019449 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.020644 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.021430 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.021513 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.021603 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.021693 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.021731 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":87,"threads_started":1}
I20260925 02:26:25.022280 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.017974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.11.125.2:37817)
0925 02:26:25.018188 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.018193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.018280 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:26:25.018351 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.018353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.018372 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.018446 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:25.018647 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.019750 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.020893 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.021563 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.021633 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.021712 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.022149 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.022197 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260925 02:26:25.022650 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.018048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:34627 (local address 127.0.0.1:57656)
0925 02:26:25.018555 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.018566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.018615 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.018864 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.018864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.018866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:25.018901 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.018903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.020113 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.020114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.021205 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.021208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.022194 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.022195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.022256 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.022259 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.022311 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.022584 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":67,"threads_started":1}
I20260925 02:26:25.022876 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.018234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.11.125.3:34627)
0925 02:26:25.018428 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.018433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.018674 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:26:25.018713 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.018715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.018729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.018795 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:25.018982 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.020027 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.021294 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.022005 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.022074 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.022160 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.022747 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.022794 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260925 02:26:25.022776 30722 tablet_service.cc:1467] Processing CreateTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 (DEFAULT_TABLE table=test-table [id=5c63ea46a63248ffa2bdeafe6614c9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:25.022990 30854 tablet_service.cc:1467] Processing CreateTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 (DEFAULT_TABLE table=test-table [id=5c63ea46a63248ffa2bdeafe6614c9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:25.023187 30722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b820e1bb7d414e96841b0f3dc6fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:25.023355 30854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b820e1bb7d414e96841b0f3dc6fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:25.023504 30986 tablet_service.cc:1467] Processing CreateTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 (DEFAULT_TABLE table=test-table [id=5c63ea46a63248ffa2bdeafe6614c9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:25.023808 30986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b820e1bb7d414e96841b0f3dc6fb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:25.026114 31066 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Bootstrap starting.
I20260925 02:26:25.026742 31066 tablet_bootstrap.cc:654] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:25.027034 31067 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Bootstrap starting.
I20260925 02:26:25.027109 31066 log.cc:824] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:25.027729 31067 tablet_bootstrap.cc:654] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:25.028061 31067 log.cc:824] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:25.028131 31066 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: No bootstrap required, opened a new log
I20260925 02:26:25.028188 31066 ts_tablet_manager.cc:1397] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:25.028409 31068 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Bootstrap starting.
I20260925 02:26:25.029141 31068 tablet_bootstrap.cc:654] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:25.029451 31068 log.cc:824] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:25.029989 31067 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: No bootstrap required, opened a new log
I20260925 02:26:25.030063 31067 ts_tablet_manager.cc:1397] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:26:25.030149 31068 tablet_bootstrap.cc:492] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: No bootstrap required, opened a new log
I20260925 02:26:25.030222 31068 ts_tablet_manager.cc:1397] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:25.031008 31066 raft_consensus.cc:348] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.031179 31066 raft_consensus.cc:374] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:25.031213 31066 raft_consensus.cc:729] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87e828eeaf040529ad6829f07272a5d, State: Initialized, Role: FOLLOWER
I20260925 02:26:25.031379 31066 consensus_queue.cc:260] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.031812 31066 ts_tablet_manager.cc:1428] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:26:25.031847 30788 heartbeater.cc:503] Master 127.11.125.62:39631 was elected leader, sending a full tablet report...
I20260925 02:26:25.032559 31067 raft_consensus.cc:348] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.032776 31067 raft_consensus.cc:374] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:25.032823 31067 raft_consensus.cc:729] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442f0c25ca534010b1290a492213884e, State: Initialized, Role: FOLLOWER
I20260925 02:26:25.032790 31068 raft_consensus.cc:348] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.032979 31068 raft_consensus.cc:374] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:25.033025 31068 raft_consensus.cc:729] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67e54a454b44930a59a45feed87dd5b, State: Initialized, Role: FOLLOWER
I20260925 02:26:25.032979 31067 consensus_queue.cc:260] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.033128 31068 consensus_queue.cc:260] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.033278 30920 heartbeater.cc:503] Master 127.11.125.62:39631 was elected leader, sending a full tablet report...
I20260925 02:26:25.033383 31068 ts_tablet_manager.cc:1428] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:25.033522 31052 heartbeater.cc:503] Master 127.11.125.62:39631 was elected leader, sending a full tablet report...
I20260925 02:26:25.033270 31067 ts_tablet_manager.cc:1428] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 02:26:25.122475 30921 tablet.cc:2369] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:25.241555 30789 tablet.cc:2369] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:25.249195 31053 tablet.cc:2369] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:25.407137 31073 raft_consensus.cc:482] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:25.407279 31073 raft_consensus.cc:504] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.407815 31073 leader_election.cc:288] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d87e828eeaf040529ad6829f07272a5d (127.11.125.1:42959), a67e54a454b44930a59a45feed87dd5b (127.11.125.3:34627)
I20260925 02:26:25.411079 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.408357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:41323 (local address 127.11.125.3:34627)
0925 02:26:25.408471 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.408473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.408483 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:25.408519 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.408520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.408527 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.408592 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:25.408767 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.409296 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410359 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.410729 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410816 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.410825 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.410981 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.411019 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:26:25.411091 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.408261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:34627 (local address 127.11.125.2:41323)
0925 02:26:25.408371 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.408399 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.408444 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.408609 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.408610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.408612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:25.408687 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.408689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.409398 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.409399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.410264 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.410266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.410817 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.410888 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.410892 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.410963 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.411002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":44,"threads_started":1}
I20260925 02:26:25.411460 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.408115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:50895 (local address 127.11.125.1:42959)
0925 02:26:25.408223 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.408226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.408258 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:26:25.408311 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.408312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.408326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.408430 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:25.408734 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.409340 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410493 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.410886 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.410987 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.411324 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.411384 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:26:25.411460 31006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b820e1bb7d414e96841b0f3dc6fb05" candidate_uuid: "442f0c25ca534010b1290a492213884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a67e54a454b44930a59a45feed87dd5b" is_pre_election: true
I20260925 02:26:25.411617 31006 raft_consensus.cc:2454] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442f0c25ca534010b1290a492213884e in term 0.
I20260925 02:26:25.411710 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.408037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42959 (local address 127.11.125.2:50895)
0925 02:26:25.408173 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.408186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:25.408246 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:25.408478 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:25.408479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:25.408482 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:25.408584 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.408586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.409430 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.409431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.410387 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:25.410390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.410979 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.410980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:25.411072 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.411076 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:25.411574 (+ 498us) client_negotiation.cc:240] Negotiation successful
0925 02:26:25.411628 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":97,"threads_started":1}
I20260925 02:26:25.411888 30809 leader_election.cc:302] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442f0c25ca534010b1290a492213884e, a67e54a454b44930a59a45feed87dd5b; no voters:
I20260925 02:26:25.412017 31073 raft_consensus.cc:2781] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:25.412082 31073 raft_consensus.cc:482] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:25.412114 31073 raft_consensus.cc:3037] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:25.412271 30742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b820e1bb7d414e96841b0f3dc6fb05" candidate_uuid: "442f0c25ca534010b1290a492213884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87e828eeaf040529ad6829f07272a5d" is_pre_election: true
I20260925 02:26:25.412386 30742 raft_consensus.cc:2454] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442f0c25ca534010b1290a492213884e in term 0.
I20260925 02:26:25.412943 31073 raft_consensus.cc:504] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.413072 31073 leader_election.cc:288] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [CANDIDATE]: Term 1 election: Requested vote from peers d87e828eeaf040529ad6829f07272a5d (127.11.125.1:42959), a67e54a454b44930a59a45feed87dd5b (127.11.125.3:34627)
I20260925 02:26:25.413288 31006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b820e1bb7d414e96841b0f3dc6fb05" candidate_uuid: "442f0c25ca534010b1290a492213884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a67e54a454b44930a59a45feed87dd5b"
I20260925 02:26:25.413303 30742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b820e1bb7d414e96841b0f3dc6fb05" candidate_uuid: "442f0c25ca534010b1290a492213884e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87e828eeaf040529ad6829f07272a5d"
I20260925 02:26:25.413391 31006 raft_consensus.cc:3037] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:25.413419 30742 raft_consensus.cc:3037] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:25.414150 31006 raft_consensus.cc:2454] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442f0c25ca534010b1290a492213884e in term 1.
I20260925 02:26:25.414343 30742 raft_consensus.cc:2454] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442f0c25ca534010b1290a492213884e in term 1.
I20260925 02:26:25.414372 30809 leader_election.cc:302] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442f0c25ca534010b1290a492213884e, a67e54a454b44930a59a45feed87dd5b; no voters:
I20260925 02:26:25.414471 31073 raft_consensus.cc:2781] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:25.414660 31073 raft_consensus.cc:686] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 1 LEADER]: Becoming Leader. State: Replica: 442f0c25ca534010b1290a492213884e, State: Running, Role: LEADER
I20260925 02:26:25.414805 31073 consensus_queue.cc:237] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } }
I20260925 02:26:25.415848 30599 catalog_manager.cc:5495] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e reported cstate change: term changed from 0 to 1, leader changed from <none> to 442f0c25ca534010b1290a492213884e (127.11.125.2). New cstate: current_term: 1 leader_uuid: "442f0c25ca534010b1290a492213884e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442f0c25ca534010b1290a492213884e" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 37817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 } health_report { overall_health: UNKNOWN } } }
I20260925 02:26:25.464671 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.462262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.11.125.2:37817)
0925 02:26:25.462329 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:25.462331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:25.462340 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:25.462417 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:25.462418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:25.462440 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:25.462509 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:25.462741 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.463215 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.464022 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:25.464204 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:25.464255 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:25.464393 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:25.464480 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:25.464551 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:25.464603 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:26:25.464612 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:26:25.466683 31006 raft_consensus.cc:1261] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 1 FOLLOWER]: Refusing update from remote peer 442f0c25ca534010b1290a492213884e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:25.466723 30742 raft_consensus.cc:1261] T c5b820e1bb7d414e96841b0f3dc6fb05 P d87e828eeaf040529ad6829f07272a5d [term 1 FOLLOWER]: Refusing update from remote peer 442f0c25ca534010b1290a492213884e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:25.467103 31077 consensus_queue.cc:1035] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a67e54a454b44930a59a45feed87dd5b" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:25.467228 31073 consensus_queue.cc:1035] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87e828eeaf040529ad6829f07272a5d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:25.470181 31082 mvcc.cc:205] Tried to move back new op lower bound from 7333081847668301824 to 7333081847459864576. Current Snapshot: MvccSnapshot[applied={T|T < 7333081847668301824}]
I20260925 02:26:25.470884 31081 mvcc.cc:205] Tried to move back new op lower bound from 7333081847668301824 to 7333081847459864576. Current Snapshot: MvccSnapshot[applied={T|T < 7333081847668301824}]
I20260925 02:26:25.473163 30599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:25.473270 30599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:25.474036 30599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9efd37d4f52348ea9fc67288a91ffb49: Sending DeleteTablet for 3 replicas of tablet c5b820e1bb7d414e96841b0f3dc6fb05
I20260925 02:26:25.474363 30722 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:25 UTC) from {username='slave'} at 127.0.0.1:56826
I20260925 02:26:25.474395 30854 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:25 UTC) from {username='slave'} at 127.0.0.1:39428
I20260925 02:26:25.474419 30986 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b820e1bb7d414e96841b0f3dc6fb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:25 UTC) from {username='slave'} at 127.0.0.1:57656
I20260925 02:26:25.474663 31091 tablet_replica.cc:330] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: stopping tablet replica
I20260925 02:26:25.474730 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30660
I20260925 02:26:25.474769 31091 raft_consensus.cc:2229] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:25.474924 31091 raft_consensus.cc:2258] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:25.475327 31089 tablet_replica.cc:330] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: stopping tablet replica
I20260925 02:26:25.475533 31089 raft_consensus.cc:2229] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:26:25.475728 31089 raft_consensus.cc:2258] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:25.476258 31089 ts_tablet_manager.cc:1905] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:25.477850 31091 ts_tablet_manager.cc:1905] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:25.478638 31089 ts_tablet_manager.cc:1918] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:25.478782 31089 log.cc:1196] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/c5b820e1bb7d414e96841b0f3dc6fb05
I20260925 02:26:25.479110 31089 ts_tablet_manager.cc:1939] T c5b820e1bb7d414e96841b0f3dc6fb05 P 442f0c25ca534010b1290a492213884e: Deleting consensus metadata
I20260925 02:26:25.480062 31091 ts_tablet_manager.cc:1918] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:25.480180 31091 log.cc:1196] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/c5b820e1bb7d414e96841b0f3dc6fb05
I20260925 02:26:25.480480 31091 ts_tablet_manager.cc:1939] T c5b820e1bb7d414e96841b0f3dc6fb05 P a67e54a454b44930a59a45feed87dd5b: Deleting consensus metadata
I20260925 02:26:25.481000 30585 catalog_manager.cc:4841] TS a67e54a454b44930a59a45feed87dd5b (127.11.125.3:34627): tablet c5b820e1bb7d414e96841b0f3dc6fb05 (table test-table [id=5c63ea46a63248ffa2bdeafe6614c9c6]) successfully deleted
I20260925 02:26:25.481992 30586 catalog_manager.cc:4841] TS 442f0c25ca534010b1290a492213884e (127.11.125.2:37817): tablet c5b820e1bb7d414e96841b0f3dc6fb05 (table test-table [id=5c63ea46a63248ffa2bdeafe6614c9c6]) successfully deleted
W20260925 02:26:25.483623 30585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:42959 (error 108)
I20260925 02:26:25.483822 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30791
I20260925 02:26:25.489159 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30923
I20260925 02:26:25.489246 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:25.489081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:42959 (local address 127.0.0.1:56838)
0925 02:26:25.489134 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:25.489191 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 02:26:25.489907 30585 catalog_manager.cc:4584] TS d87e828eeaf040529ad6829f07272a5d (127.11.125.1:42959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5b820e1bb7d414e96841b0f3dc6fb05: Network error: Client connection negotiation failed: client connection to 127.11.125.1:42959: connect: Connection refused (error 111)
I20260925 02:26:25.494318 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 30568
2026-09-25T02:26:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1101 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:25 dist-test-slave-hj7s krb5kdc[31095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:25 dist-test-slave-hj7s krb5kdc[31095](info): set up 2 sockets
Sep 25 02:26:25 dist-test-slave-hj7s krb5kdc[31095](info): commencing operation
krb5kdc: starting...
W20260925 02:26:26.571120 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:26 dist-test-slave-hj7s krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:26.692305 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38513
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:26.773500 31108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:26.773702 31108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:26.773725 31108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:26.774799 31108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:26.774834 31108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:26.774856 31108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:26.774869 31108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:26.774881 31108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:26.776273 31108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:26.776532 31108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:26.776782 31108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:26.781709 31116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:26.781716 31113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:26.781744 31108 server_base.cc:1034] running on GCE node
W20260925 02:26:26.781716 31114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:26.782198 31108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:26.782477 31108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:26.783646 31108 hybrid_clock.cc:648] HybridClock initialized: now 1790303186783630 us; error 26 us; skew 500 ppm
Sep 25 02:26:26 dist-test-slave-hj7s krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303186, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:26.786262 31108 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:26.786736 31108 webserver.cc:460] Webserver started at http://127.11.125.62:42897/ using document root <none> and password file <none>
I20260925 02:26:26.786963 31108 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:26.787034 31108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:26.787190 31108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:26.788237 31108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "55c98f399edb4d0ea91e50252b6ccc8e"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "2f949a79c6ce1540fe57f003d7a66413"
server_key_iv: "0c3cbe33546d5c1dcde00e1c73b19474"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.788614 31108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "55c98f399edb4d0ea91e50252b6ccc8e"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "2f949a79c6ce1540fe57f003d7a66413"
server_key_iv: "0c3cbe33546d5c1dcde00e1c73b19474"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.790007 31108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:26:26.790747 31123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:26.790911 31108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:26.790997 31108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "55c98f399edb4d0ea91e50252b6ccc8e"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "2f949a79c6ce1540fe57f003d7a66413"
server_key_iv: "0c3cbe33546d5c1dcde00e1c73b19474"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.791067 31108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:26.806897 31108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:26.809258 31108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:26.809420 31108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:26.813380 31108 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:38513
I20260925 02:26:26.813453 31175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:38513 every 8 connection(s)
I20260925 02:26:26.813769 31108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:26.814642 31176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:26.817010 31176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: Bootstrap starting.
I20260925 02:26:26.817040 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31108
I20260925 02:26:26.817178 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:26.817387 11764 external_mini_cluster.cc:1428] Setting key 05beb053ece43f6ad47dda29fd8c4e39
I20260925 02:26:26.817744 31176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:26.818328 31176 log.cc:824] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:26.819092 31176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: No bootstrap required, opened a new log
I20260925 02:26:26.821460 31176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } }
I20260925 02:26:26.821615 31176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:26.821666 31176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c98f399edb4d0ea91e50252b6ccc8e, State: Initialized, Role: FOLLOWER
I20260925 02:26:26.821797 31176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } }
I20260925 02:26:26.821888 31176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:26.821947 31176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:26.822000 31176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:26.822525 31176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } }
I20260925 02:26:26.822649 31176 leader_election.cc:302] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55c98f399edb4d0ea91e50252b6ccc8e; no voters:
I20260925 02:26:26.822826 31176 leader_election.cc:288] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:26.822866 31182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:26.823099 31182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [term 1 LEADER]: Becoming Leader. State: Replica: 55c98f399edb4d0ea91e50252b6ccc8e, State: Running, Role: LEADER
I20260925 02:26:26.823112 31176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:26.823299 31182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } }
I20260925 02:26:26.824508 31184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c98f399edb4d0ea91e50252b6ccc8e. Latest consensus state: current_term: 1 leader_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } } }
Sep 25 02:26:26 dist-test-slave-hj7s krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:26.824519 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c98f399edb4d0ea91e50252b6ccc8e" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 38513 } } }
I20260925 02:26:26.824610 31184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:26.825133 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e [sys.catalog]: This master's current role is: LEADER
W20260925 02:26:26.825335 31196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:26.825397 31196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:26.825459 31199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:26.825655 31199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:26.827450 31199 catalog_manager.cc:1347] Generated new cluster ID: f02ced1a521f49f89b68bdd28f4e580f
I20260925 02:26:26.827570 31199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:26.829131 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:26.817916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.11.125.62:38513)
0925 02:26:26.818098 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:26.818109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:26.818148 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:26:26.818725 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:26.818754 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:26.818819 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:26.818969 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:26.819618 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:26.821103 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:26.821960 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:26.822431 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:26.822549 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:26.825394 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:26.825397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:26.825398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:26.825409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:26.827427 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:26.827904 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:26.827907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:26.827907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:26.828024 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:26.828242 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:26.828243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:26.828243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:26.828482 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:26.828667 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:26.828750 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:26:26.828930 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260925 02:26:26.850462 31199 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:26.851205 31199 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:26.855122 31199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: Generated new TSK 0
I20260925 02:26:26.855358 31199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.125.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
Entry for principal kudu/127.11.125.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab.
I20260925 02:26:26.883140 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:26.966451 31203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:26.966674 31203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:26.966701 31203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:26.967948 31203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:26.967995 31203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:26.968024 31203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:26.969549 31203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.1.keytab
--principal=kudu/127.11.125.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:26.969805 31203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:26.970067 31203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:26.975692 31211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:26.975692 31209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:26.975790 31203 server_base.cc:1034] running on GCE node
W20260925 02:26:26.975728 31208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:26.976156 31203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:26.976444 31203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:26.977612 31203 hybrid_clock.cc:648] HybridClock initialized: now 1790303186977592 us; error 50 us; skew 500 ppm
Sep 25 02:26:26 dist-test-slave-hj7s krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303186, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:26.980036 31203 init.cc:377] Logged in from keytab as kudu/127.11.125.1@KRBTEST.COM (short username kudu)
I20260925 02:26:26.980481 31203 webserver.cc:460] Webserver started at http://127.11.125.1:34183/ using document root <none> and password file <none>
I20260925 02:26:26.980712 31203 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:26.980783 31203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:26.980967 31203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:26.982034 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "6d5edaee08a04f0987af5c69dc79b70d"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "899091f5f55de48f7e35f23d163374ca"
server_key_iv: "a6c127f3ccdd95adf4c202878f8b145e"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.982412 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "6d5edaee08a04f0987af5c69dc79b70d"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "899091f5f55de48f7e35f23d163374ca"
server_key_iv: "a6c127f3ccdd95adf4c202878f8b145e"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.983876 31203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:26.984622 31219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:26.984799 31203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:26.984907 31203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "6d5edaee08a04f0987af5c69dc79b70d"
format_stamp: "Formatted at 2026-09-25 02:26:26 on dist-test-slave-hj7s"
server_key: "899091f5f55de48f7e35f23d163374ca"
server_key_iv: "a6c127f3ccdd95adf4c202878f8b145e"
server_key_version: "encryptionkey@0"
I20260925 02:26:26.985023 31203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:27.004112 31203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:27.006068 31203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:27.006237 31203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:27.006484 31203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:27.006800 31203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:27.006855 31203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.006892 31203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:27.006940 31203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.012744 31203 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:45871
I20260925 02:26:27.012789 31333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:45871 every 8 connection(s)
I20260925 02:26:27.013150 31203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:27.018215 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31203
I20260925 02:26:27.018371 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:27.018617 11764 external_mini_cluster.cc:1428] Setting key a3babbdfdf77cea5541fd8173c195ee0
Sep 25 02:26:27 dist-test-slave-hj7s krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303186, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.1@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:27.023413 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.013812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:50369 (local address 127.11.125.62:38513)
0925 02:26:27.013856 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.013858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.014108 (+ 250us) server_negotiation.cc:407] Connection header received
0925 02:26:27.014645 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.014647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.014660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.014726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:27.015747 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.016253 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.017327 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.017543 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.017621 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.020078 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:27.020079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:27.020080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:27.020088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:27.021753 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.022235 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.022237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.022237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.022373 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.022666 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.022667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.022667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.022816 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:27.022888 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.023210 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.023333 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:26:27.023693 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.013795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38513 (local address 127.11.125.1:50369)
0925 02:26:27.013978 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.014015 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.014537 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.014811 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.014815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.015452 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:27.015642 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.015648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.016353 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.016354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.017237 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.017240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.017631 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.017632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.017756 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.018285 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:27.018296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:27.019988 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:27.021820 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.021828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.021830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.022164 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.022451 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.022452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.022452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.022606 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.022886 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:27.022894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:27.023074 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.023197 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.023401 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 02:26:27.024248 31334 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38513
I20260925 02:26:27.024363 31334 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:27.024547 31334 heartbeater.cc:511] Master 127.11.125.62:38513 requested a full tablet report, sending...
I20260925 02:26:27.025171 31140 ts_manager.cc:194] Registered new tserver with Master: 6d5edaee08a04f0987af5c69dc79b70d (127.11.125.1:45871)
I20260925 02:26:27.025790 31140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.1@KRBTEST.COM'} at 127.11.125.1:50369
WARNING: no policy specified for kudu/127.11.125.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
Entry for principal kudu/127.11.125.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab.
I20260925 02:26:27.043742 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:27.125537 31339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:27.125758 31339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:27.125794 31339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:27.126928 31339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:27.126989 31339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:27.127034 31339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:27.128479 31339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.2.keytab
--principal=kudu/127.11.125.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:27.128743 31339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:27.128997 31339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:27.134203 31344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:27.134194 31347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.134243 31339 server_base.cc:1034] running on GCE node
W20260925 02:26:27.134178 31345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.134616 31339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:27.134830 31339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:27.135977 31339 hybrid_clock.cc:648] HybridClock initialized: now 1790303187135962 us; error 31 us; skew 500 ppm
Sep 25 02:26:27 dist-test-slave-hj7s krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303187, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:27.138420 31339 init.cc:377] Logged in from keytab as kudu/127.11.125.2@KRBTEST.COM (short username kudu)
I20260925 02:26:27.138875 31339 webserver.cc:460] Webserver started at http://127.11.125.2:35151/ using document root <none> and password file <none>
I20260925 02:26:27.139071 31339 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:27.139111 31339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:27.139199 31339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:27.140183 31339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "39f7feda533528f1ea08f80ff384e9e1"
server_key_iv: "08e3d23df23af621aae74365e225fccf"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.140475 31339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "39f7feda533528f1ea08f80ff384e9e1"
server_key_iv: "08e3d23df23af621aae74365e225fccf"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.141810 31339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 02:26:27.142531 31354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.142710 31339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:27.142813 31339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "39f7feda533528f1ea08f80ff384e9e1"
server_key_iv: "08e3d23df23af621aae74365e225fccf"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.142901 31339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:27.167943 31339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:27.169889 31339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:27.170055 31339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:27.170295 31339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:27.170609 31339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:27.170661 31339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.170696 31339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:27.170740 31339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.176637 31339 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:36795
I20260925 02:26:27.176682 31468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:36795 every 8 connection(s)
I20260925 02:26:27.177073 31339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:26:27.178472 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31339
I20260925 02:26:27.178594 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:27.178798 11764 external_mini_cluster.cc:1428] Setting key 13ddd4f0791f02dbc022d225d9aec3cb
Sep 25 02:26:27 dist-test-slave-hj7s krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303187, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.2@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.3@KRBTEST.COM; defaulting to no policy
I20260925 02:26:27.188004 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.177796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:49651 (local address 127.11.125.62:38513)
0925 02:26:27.177878 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.177880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.178068 (+ 188us) server_negotiation.cc:407] Connection header received
0925 02:26:27.178756 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.178757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.178769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.178814 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:27.180141 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.180676 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.181897 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.182212 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.182289 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.184957 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:27.184958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:27.184958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:27.184965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:27.186329 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.186850 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.186851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.186852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.186904 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.187229 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.187230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.187230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.187358 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:27.187413 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.187863 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.187935 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:26:27.188262 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.177749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38513 (local address 127.11.125.2:49651)
0925 02:26:27.177938 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.177977 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.178649 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.178902 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.178907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.179730 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:27.180045 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.180052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.180845 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.180846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.181803 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.181806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.182302 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.182302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.182412 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.182964 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:27.182972 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:27.184815 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:27.186436 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.186444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.186447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.186769 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.186973 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.186974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.186974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.187132 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.187428 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:27.187435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:27.187673 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.187824 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.188010 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260925 02:26:27.188791 31469 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38513
I20260925 02:26:27.188941 31469 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:27.189154 31469 heartbeater.cc:511] Master 127.11.125.62:38513 requested a full tablet report, sending...
I20260925 02:26:27.189644 31140 ts_manager.cc:194] Registered new tserver with Master: a9ddf8c50d4a4218bb8dfec493f5d62b (127.11.125.2:36795)
I20260925 02:26:27.190032 31140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.2@KRBTEST.COM'} at 127.11.125.2:49651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
Entry for principal kudu/127.11.125.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab.
I20260925 02:26:27.206183 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:27.286777 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:27.287015 31474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:27.287057 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:27.288261 31474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:27.288319 31474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:27.288360 31474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:27.289873 31474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.3.keytab
--principal=kudu/127.11.125.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:27.290136 31474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:27.290357 31474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:27.295517 31480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:27.295552 31482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.295651 31474 server_base.cc:1034] running on GCE node
W20260925 02:26:27.295539 31479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.295953 31474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:27.296191 31474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:27.297353 31474 hybrid_clock.cc:648] HybridClock initialized: now 1790303187297335 us; error 43 us; skew 500 ppm
Sep 25 02:26:27 dist-test-slave-hj7s krb5kdc[31095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303187, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:27.299853 31474 init.cc:377] Logged in from keytab as kudu/127.11.125.3@KRBTEST.COM (short username kudu)
I20260925 02:26:27.300349 31474 webserver.cc:460] Webserver started at http://127.11.125.3:34635/ using document root <none> and password file <none>
I20260925 02:26:27.300592 31474 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:27.300676 31474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:27.300829 31474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:27.301882 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "443055c8c72348d1b695a12867bdf06c"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "84229ca40cb0589034a5563811630536"
server_key_iv: "96fd360adfefea710862ac23bd384160"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.302235 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "443055c8c72348d1b695a12867bdf06c"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "84229ca40cb0589034a5563811630536"
server_key_iv: "96fd360adfefea710862ac23bd384160"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.303520 31474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:27.304263 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.304435 31474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:27.304540 31474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "443055c8c72348d1b695a12867bdf06c"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "84229ca40cb0589034a5563811630536"
server_key_iv: "96fd360adfefea710862ac23bd384160"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.304626 31474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:27.322299 31474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:27.324450 31474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:27.324630 31474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:27.325027 31474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:27.325385 31474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:27.325441 31474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.325485 31474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:27.325512 31474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.331354 31474 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:39997
I20260925 02:26:27.331467 31603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:39997 every 8 connection(s)
I20260925 02:26:27.331784 31474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:26:27 dist-test-slave-hj7s krb5kdc[31095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303187, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.3@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:27.340641 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.332529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:39479 (local address 127.11.125.62:38513)
0925 02:26:27.332599 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.332601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.332842 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:26:27.333377 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.333378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.333389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.333443 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:27.334365 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.334855 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.335778 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.335958 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.336008 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.338211 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:27.338212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:27.338213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:27.338219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:27.339460 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.339851 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.339852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.339853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.339906 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.340124 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.340124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.340125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.340228 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:27.340279 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.340509 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.340572 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:26:27.340802 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.332524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:38513 (local address 127.11.125.3:39479)
0925 02:26:27.332714 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.332751 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.333288 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.333498 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.333501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.334088 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:27.334283 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.334289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.334938 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.334939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.335705 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.335707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.336030 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.336030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.336134 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.336665 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:27.336674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:27.338112 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:27.339532 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.339539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.339540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.339781 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.339968 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:27.339969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:27.339969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:27.340068 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:27.340282 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:27.340286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:27.340401 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.340481 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.340602 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260925 02:26:27.340957 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31474
I20260925 02:26:27.341079 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:27.341271 11764 external_mini_cluster.cc:1428] Setting key ae08b68e269a72ba1e8f7c123b492f1c
I20260925 02:26:27.341460 31604 heartbeater.cc:347] Connected to a master server at 127.11.125.62:38513
I20260925 02:26:27.341585 31604 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:27.341784 31604 heartbeater.cc:511] Master 127.11.125.62:38513 requested a full tablet report, sending...
I20260925 02:26:27.342192 31140 ts_manager.cc:194] Registered new tserver with Master: 443055c8c72348d1b695a12867bdf06c (127.11.125.3:39997)
I20260925 02:26:27.342595 31140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.125.3@KRBTEST.COM'} at 127.11.125.3:39479
I20260925 02:26:27.353390 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:27.361090 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.354747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.11.125.62:38513)
0925 02:26:27.354825 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.354827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.354880 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:26:27.354913 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.354914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.354926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.354995 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:27.355601 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.356134 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.357215 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.357398 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.357452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.358765 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:27.358766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:27.358766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:27.358771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:27.359994 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.360315 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.360316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.360317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.360366 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:27.360526 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:27.360527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:27.360527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:27.360667 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:27.360747 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.360859 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.360979 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:26:27.362839 31140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:27.364456 31140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:27.370990 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.366846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:45871 (local address 127.0.0.1:39566)
0925 02:26:27.367114 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.367124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.367168 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.367482 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.367485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.367488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:27.367560 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.367561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.368601 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.368602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.369877 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.369880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.370669 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.370758 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.370773 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.370846 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.370911 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":6,"thread_start_us":104,"threads_started":1}
I20260925 02:26:27.370996 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.367112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.11.125.1:45871)
0925 02:26:27.367309 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.367313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.367322 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:27.367375 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.367377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.367390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.367464 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:27.367653 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.368503 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.369969 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.370581 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370670 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.370737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.370869 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.370915 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260925 02:26:27.371327 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.366818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36795 (local address 127.0.0.1:54646)
0925 02:26:27.366998 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.367039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.367107 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.367791 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.367792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.367795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:27.367846 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.367848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.369146 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.369148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.370133 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.370136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.371070 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.371072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.371157 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.371161 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.371219 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.371259 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
I20260925 02:26:27.371409 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.366930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.11.125.2:36795)
0925 02:26:27.367626 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.367630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.367640 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:27.367679 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.367681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.367698 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.367777 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:27.367920 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.369052 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370210 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.370910 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370973 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.371031 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.371264 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.371317 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":97,"threads_started":1}
I20260925 02:26:27.371896 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.366846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39997 (local address 127.0.0.1:46150)
0925 02:26:27.367219 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.367227 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.367260 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.367555 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.367557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.367559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:27.367616 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.367618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.369006 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.369007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.370079 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.370081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.371382 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.371383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.371717 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.371723 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.371780 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.371819 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260925 02:26:27.371919 31267 tablet_service.cc:1467] Processing CreateTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 (DEFAULT_TABLE table=test-table [id=1d9d34a8825345ec96c284342fd96000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:27.372193 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.367098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.11.125.3:39997)
0925 02:26:27.367296 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.367302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.367314 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:26:27.367358 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.367360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.367376 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.367464 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:27.367704 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.368873 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370159 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.370898 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.370973 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.371031 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.372061 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.372111 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260925 02:26:27.372133 31403 tablet_service.cc:1467] Processing CreateTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 (DEFAULT_TABLE table=test-table [id=1d9d34a8825345ec96c284342fd96000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:27.372375 31267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f403dbcf8304d90890d7b94b97c3fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:27.372496 31403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f403dbcf8304d90890d7b94b97c3fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:27.372772 31538 tablet_service.cc:1467] Processing CreateTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 (DEFAULT_TABLE table=test-table [id=1d9d34a8825345ec96c284342fd96000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:27.373113 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f403dbcf8304d90890d7b94b97c3fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:27.375162 31621 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Bootstrap starting.
I20260925 02:26:27.375950 31621 tablet_bootstrap.cc:654] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:27.375924 31622 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Bootstrap starting.
I20260925 02:26:27.376422 31623 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Bootstrap starting.
I20260925 02:26:27.376760 31622 tablet_bootstrap.cc:654] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:27.376744 31621 log.cc:824] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:27.377115 31623 tablet_bootstrap.cc:654] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:27.377405 31622 log.cc:824] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:27.377681 31621 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: No bootstrap required, opened a new log
I20260925 02:26:27.377750 31623 log.cc:824] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:27.377790 31621 ts_tablet_manager.cc:1397] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:26:27.378525 31623 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: No bootstrap required, opened a new log
I20260925 02:26:27.378554 31622 tablet_bootstrap.cc:492] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: No bootstrap required, opened a new log
I20260925 02:26:27.378597 31623 ts_tablet_manager.cc:1397] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:27.378612 31622 ts_tablet_manager.cc:1397] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:26:27.379909 31623 raft_consensus.cc:348] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.380079 31623 raft_consensus.cc:374] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:27.380125 31623 raft_consensus.cc:729] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443055c8c72348d1b695a12867bdf06c, State: Initialized, Role: FOLLOWER
I20260925 02:26:27.379998 31621 raft_consensus.cc:348] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.380162 31621 raft_consensus.cc:374] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:27.380208 31621 raft_consensus.cc:729] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5edaee08a04f0987af5c69dc79b70d, State: Initialized, Role: FOLLOWER
I20260925 02:26:27.380260 31623 consensus_queue.cc:260] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.380343 31621 consensus_queue.cc:260] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.380551 31623 ts_tablet_manager.cc:1428] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:26:27.380625 31621 ts_tablet_manager.cc:1428] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:26:27.380676 31604 heartbeater.cc:503] Master 127.11.125.62:38513 was elected leader, sending a full tablet report...
I20260925 02:26:27.380698 31334 heartbeater.cc:503] Master 127.11.125.62:38513 was elected leader, sending a full tablet report...
I20260925 02:26:27.380620 31622 raft_consensus.cc:348] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.381101 31622 raft_consensus.cc:374] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:27.381150 31622 raft_consensus.cc:729] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddf8c50d4a4218bb8dfec493f5d62b, State: Initialized, Role: FOLLOWER
I20260925 02:26:27.381278 31622 consensus_queue.cc:260] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.381541 31622 ts_tablet_manager.cc:1428] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:26:27.381673 31469 heartbeater.cc:503] Master 127.11.125.62:38513 was elected leader, sending a full tablet report...
W20260925 02:26:27.427768 31470 tablet.cc:2369] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:27.513898 31335 tablet.cc:2369] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:26:27.582389 31605 tablet.cc:2369] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:27.612138 31628 raft_consensus.cc:482] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:27.612310 31628 raft_consensus.cc:504] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.612869 31628 leader_election.cc:288] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddf8c50d4a4218bb8dfec493f5d62b (127.11.125.2:36795), 443055c8c72348d1b695a12867bdf06c (127.11.125.3:39997)
I20260925 02:26:27.617254 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.613157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:59559 (local address 127.11.125.3:39997)
0925 02:26:27.613250 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.613252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.613335 (+ 83us) server_negotiation.cc:407] Connection header received
0925 02:26:27.613396 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.613397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.613408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.613484 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:27.613744 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.614528 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616159 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.616798 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.616922 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.617139 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.617185 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:26:27.617254 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.613178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:41893 (local address 127.11.125.2:36795)
0925 02:26:27.613243 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.613245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.613335 (+ 90us) server_negotiation.cc:407] Connection header received
0925 02:26:27.613396 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.613397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.613409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.613491 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:27.613744 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.614513 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616159 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.616798 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.616926 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.617133 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.617184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:26:27.617244 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.613105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:36795 (local address 127.11.125.1:41893)
0925 02:26:27.613250 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.613261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.613331 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.613522 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.613523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.613526 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:27.613655 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.613657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.614610 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.614611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.616025 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.616029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.616925 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.617009 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.617017 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.617088 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.617131 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":135,"thread_start_us":66,"threads_started":1}
I20260925 02:26:27.617252 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.613102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:39997 (local address 127.11.125.1:59559)
0925 02:26:27.613247 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.613261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:27.613331 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:27.613522 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:27.613523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:27.613527 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:27.613646 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.613649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.614610 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.614611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.616025 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:27.616029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.616927 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.616928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:27.617009 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.617016 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:26:27.617078 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:26:27.617131 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":81,"thread_start_us":84,"threads_started":1}
I20260925 02:26:27.617760 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f403dbcf8304d90890d7b94b97c3fb6" candidate_uuid: "6d5edaee08a04f0987af5c69dc79b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443055c8c72348d1b695a12867bdf06c" is_pre_election: true
I20260925 02:26:27.617916 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f403dbcf8304d90890d7b94b97c3fb6" candidate_uuid: "6d5edaee08a04f0987af5c69dc79b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" is_pre_election: true
I20260925 02:26:27.617962 31558 raft_consensus.cc:2454] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5edaee08a04f0987af5c69dc79b70d in term 0.
I20260925 02:26:27.618054 31423 raft_consensus.cc:2454] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5edaee08a04f0987af5c69dc79b70d in term 0.
I20260925 02:26:27.618183 31221 leader_election.cc:302] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443055c8c72348d1b695a12867bdf06c, 6d5edaee08a04f0987af5c69dc79b70d; no voters:
I20260925 02:26:27.618337 31628 raft_consensus.cc:2781] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:27.618412 31628 raft_consensus.cc:482] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:27.618443 31628 raft_consensus.cc:3037] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:27.619167 31628 raft_consensus.cc:504] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.619316 31628 leader_election.cc:288] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [CANDIDATE]: Term 1 election: Requested vote from peers a9ddf8c50d4a4218bb8dfec493f5d62b (127.11.125.2:36795), 443055c8c72348d1b695a12867bdf06c (127.11.125.3:39997)
I20260925 02:26:27.619460 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f403dbcf8304d90890d7b94b97c3fb6" candidate_uuid: "6d5edaee08a04f0987af5c69dc79b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b"
I20260925 02:26:27.619515 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f403dbcf8304d90890d7b94b97c3fb6" candidate_uuid: "6d5edaee08a04f0987af5c69dc79b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443055c8c72348d1b695a12867bdf06c"
I20260925 02:26:27.619549 31423 raft_consensus.cc:3037] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:27.619580 31558 raft_consensus.cc:3037] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:27.620481 31558 raft_consensus.cc:2454] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5edaee08a04f0987af5c69dc79b70d in term 1.
I20260925 02:26:27.620481 31423 raft_consensus.cc:2454] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5edaee08a04f0987af5c69dc79b70d in term 1.
I20260925 02:26:27.620700 31223 leader_election.cc:302] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5edaee08a04f0987af5c69dc79b70d, a9ddf8c50d4a4218bb8dfec493f5d62b; no voters:
I20260925 02:26:27.620813 31628 raft_consensus.cc:2781] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:27.621016 31628 raft_consensus.cc:686] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [term 1 LEADER]: Becoming Leader. State: Replica: 6d5edaee08a04f0987af5c69dc79b70d, State: Running, Role: LEADER
I20260925 02:26:27.621150 31628 consensus_queue.cc:237] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } }
I20260925 02:26:27.622078 31138 catalog_manager.cc:5495] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d5edaee08a04f0987af5c69dc79b70d (127.11.125.1). New cstate: current_term: 1 leader_uuid: "6d5edaee08a04f0987af5c69dc79b70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5edaee08a04f0987af5c69dc79b70d" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 45871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260925 02:26:27.657608 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.655138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.11.125.1:45871)
0925 02:26:27.655206 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.655208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.655218 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:27.655274 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.655275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.655300 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.655380 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:27.655536 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.656013 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.656789 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.657016 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.657096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.657268 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:27.657372 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:27.657470 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.657489 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.657511 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:26:27.659531 31558 raft_consensus.cc:1261] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 1 FOLLOWER]: Refusing update from remote peer 6d5edaee08a04f0987af5c69dc79b70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:27.659648 31423 raft_consensus.cc:1261] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 1 FOLLOWER]: Refusing update from remote peer 6d5edaee08a04f0987af5c69dc79b70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:27.659883 31628 consensus_queue.cc:1035] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "443055c8c72348d1b695a12867bdf06c" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:27.660014 31632 consensus_queue.cc:1035] T 3f403dbcf8304d90890d7b94b97c3fb6 P 6d5edaee08a04f0987af5c69dc79b70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ddf8c50d4a4218bb8dfec493f5d62b" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:27.662828 31637 mvcc.cc:205] Tried to move back new op lower bound from 7333081856650502144 to 7333081856496943104. Current Snapshot: MvccSnapshot[applied={T|T < 7333081856650502144}]
I20260925 02:26:27.663020 31635 mvcc.cc:205] Tried to move back new op lower bound from 7333081856650502144 to 7333081856496943104. Current Snapshot: MvccSnapshot[applied={T|T < 7333081856650502144}]
I20260925 02:26:27.665956 31138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:27.666064 31138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:27.666785 31138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55c98f399edb4d0ea91e50252b6ccc8e: Sending DeleteTablet for 3 replicas of tablet 3f403dbcf8304d90890d7b94b97c3fb6
I20260925 02:26:27.667121 31403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:27 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:54646
I20260925 02:26:27.667160 31267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:27 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260925 02:26:27.667142 31538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f403dbcf8304d90890d7b94b97c3fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:27 UTC) from {username='kudu', principal='kudu/127.11.125.62@KRBTEST.COM'} at 127.0.0.1:46150
I20260925 02:26:27.667428 31648 tablet_replica.cc:330] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: stopping tablet replica
I20260925 02:26:27.667521 31646 tablet_replica.cc:330] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: stopping tablet replica
I20260925 02:26:27.667572 31648 raft_consensus.cc:2229] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:27.667690 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31203
I20260925 02:26:27.667729 31648 raft_consensus.cc:2258] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:27.667816 31646 raft_consensus.cc:2229] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:27.667922 31646 raft_consensus.cc:2258] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:27.668246 31648 ts_tablet_manager.cc:1905] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:27.670252 31648 ts_tablet_manager.cc:1918] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:27.670320 31648 log.cc:1196] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/3f403dbcf8304d90890d7b94b97c3fb6
I20260925 02:26:27.670590 31648 ts_tablet_manager.cc:1939] T 3f403dbcf8304d90890d7b94b97c3fb6 P 443055c8c72348d1b695a12867bdf06c: Deleting consensus metadata
I20260925 02:26:27.671006 31125 catalog_manager.cc:4841] TS 443055c8c72348d1b695a12867bdf06c (127.11.125.3:39997): tablet 3f403dbcf8304d90890d7b94b97c3fb6 (table test-table [id=1d9d34a8825345ec96c284342fd96000]) successfully deleted
I20260925 02:26:27.672200 31646 ts_tablet_manager.cc:1905] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:27.674142 31646 ts_tablet_manager.cc:1918] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:27.674227 31646 log.cc:1196] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/3f403dbcf8304d90890d7b94b97c3fb6
I20260925 02:26:27.674482 31646 ts_tablet_manager.cc:1939] T 3f403dbcf8304d90890d7b94b97c3fb6 P a9ddf8c50d4a4218bb8dfec493f5d62b: Deleting consensus metadata
I20260925 02:26:27.675097 31127 catalog_manager.cc:4841] TS a9ddf8c50d4a4218bb8dfec493f5d62b (127.11.125.2:36795): tablet 3f403dbcf8304d90890d7b94b97c3fb6 (table test-table [id=1d9d34a8825345ec96c284342fd96000]) successfully deleted
W20260925 02:26:27.676476 31126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:45871 (error 108)
I20260925 02:26:27.676607 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31339
I20260925 02:26:27.677398 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.676956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:45871 (local address 127.0.0.1:39570)
0925 02:26:27.677331 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:27.677369 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:45871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355}
W20260925 02:26:27.677567 31126 catalog_manager.cc:4584] TS 6d5edaee08a04f0987af5c69dc79b70d (127.11.125.1:45871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f403dbcf8304d90890d7b94b97c3fb6: Network error: Client connection negotiation failed: client connection to 127.11.125.1:45871: connect: Connection refused (error 111)
I20260925 02:26:27.683867 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31474
I20260925 02:26:27.689975 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31108
2026-09-25T02:26:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:27Z Disabled control of system clock
I20260925 02:26:27.717599 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:46445
--webserver_interface=127.11.125.62
--webserver_port=0
--builtin_ntp_servers=127.11.125.20:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:27.800333 31653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:27.800539 31653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:27.800560 31653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:27.801791 31653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:27.801833 31653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:27.801846 31653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:27.801859 31653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:27.801872 31653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:27.803129 31653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:46445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:27.803310 31653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:27.803530 31653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:27.808314 31661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:27.808343 31659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.808403 31653 server_base.cc:1034] running on GCE node
W20260925 02:26:27.808343 31658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.808820 31653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:27.809147 31653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:27.810307 31653 hybrid_clock.cc:648] HybridClock initialized: now 1790303187810290 us; error 27 us; skew 500 ppm
I20260925 02:26:27.811537 31653 webserver.cc:460] Webserver started at http://127.11.125.62:35735/ using document root <none> and password file <none>
I20260925 02:26:27.811777 31653 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:27.811822 31653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:27.812011 31653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:27.813162 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "65cd744e7e3a48dda5419396664d89e1"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "194002db708039f516d733cfea8a811a"
server_key_iv: "d4557a245edfb800e672a6a33adf0dab"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.813534 31653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "65cd744e7e3a48dda5419396664d89e1"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "194002db708039f516d733cfea8a811a"
server_key_iv: "d4557a245edfb800e672a6a33adf0dab"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.814821 31653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:27.815565 31667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.815729 31653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:27.815838 31653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "65cd744e7e3a48dda5419396664d89e1"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "194002db708039f516d733cfea8a811a"
server_key_iv: "d4557a245edfb800e672a6a33adf0dab"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.815927 31653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:27.825388 31653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:27.825670 31653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:27.825821 31653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:27.829875 31653 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:46445
I20260925 02:26:27.829907 31719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:46445 every 8 connection(s)
I20260925 02:26:27.830229 31653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:27.831151 31720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:27.831583 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31653
I20260925 02:26:27.831679 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:27.831899 11764 external_mini_cluster.cc:1428] Setting key 336a28f15aaa13df3cfd19e5c0a0ab30
I20260925 02:26:27.834270 31720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: Bootstrap starting.
I20260925 02:26:27.834972 31720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:27.835268 31720 log.cc:824] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:27.835954 31720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: No bootstrap required, opened a new log
I20260925 02:26:27.838073 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:27.832498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.11.125.62:46445)
0925 02:26:27.832731 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:27.832740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:27.832796 (+ 56us) server_negotiation.cc:407] Connection header received
0925 02:26:27.833401 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:27.833410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:27.833491 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:27.833639 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:27.833909 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.835221 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.836083 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:27.836513 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:27.836653 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:27.837438 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:27.837442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:27.837442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:27.837619 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:27.837652 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:27.837662 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:27.837743 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:27.837813 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:26:27.837901 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260925 02:26:27.838066 31720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } }
I20260925 02:26:27.838261 31720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:27.838296 31720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cd744e7e3a48dda5419396664d89e1, State: Initialized, Role: FOLLOWER
I20260925 02:26:27.838431 31720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } }
I20260925 02:26:27.838531 31720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:27.838578 31720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:27.838626 31720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:27.839188 31720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } }
I20260925 02:26:27.839308 31720 leader_election.cc:302] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65cd744e7e3a48dda5419396664d89e1; no voters:
I20260925 02:26:27.839505 31720 leader_election.cc:288] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:27.839560 31725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:27.839709 31725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [term 1 LEADER]: Becoming Leader. State: Replica: 65cd744e7e3a48dda5419396664d89e1, State: Running, Role: LEADER
I20260925 02:26:27.839778 31720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:27.839839 31725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } }
I20260925 02:26:27.840201 31727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65cd744e7e3a48dda5419396664d89e1. Latest consensus state: current_term: 1 leader_uuid: "65cd744e7e3a48dda5419396664d89e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } } }
I20260925 02:26:27.840286 31727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:27.840487 31726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65cd744e7e3a48dda5419396664d89e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65cd744e7e3a48dda5419396664d89e1" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 46445 } } }
I20260925 02:26:27.840553 31726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:27.840858 31733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:27.841406 31733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:27.842935 31733 catalog_manager.cc:1347] Generated new cluster ID: b8736aab5cb947db964cc55a33933dad
I20260925 02:26:27.842993 31733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:27.852325 31733 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:27.853094 31733 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:27.863914 31733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: Generated new TSK 0
I20260925 02:26:27.864118 31733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:27.869225 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.1:0
--local_ip_for_outbound_sockets=127.11.125.1
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--builtin_ntp_servers=127.11.125.20:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:27.971820 31744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:27.972050 31744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:27.972077 31744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:27.973476 31744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:27.973528 31744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:27.973559 31744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.1
I20260925 02:26:27.975132 31744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.125.1
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.1
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:27.975430 31744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:27.975812 31744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:27.982360 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:27.982363 31750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:27.982493 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:27.982692 31744 server_base.cc:1034] running on GCE node
I20260925 02:26:27.982947 31744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:27.983203 31744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:27.984373 31744 hybrid_clock.cc:648] HybridClock initialized: now 1790303187984359 us; error 30 us; skew 500 ppm
I20260925 02:26:27.985762 31744 webserver.cc:460] Webserver started at http://127.11.125.1:33321/ using document root <none> and password file <none>
I20260925 02:26:27.986025 31744 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:27.986119 31744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:27.986256 31744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:27.987432 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data/instance:
uuid: "bf7515477d0144fabafabe4c6aa8affa"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "0bbdd4623dcfcc083d6d6bebb0e4802d"
server_key_iv: "ece3438d6ab38a869ea3b89b3ca355da"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.987804 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance:
uuid: "bf7515477d0144fabafabe4c6aa8affa"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "0bbdd4623dcfcc083d6d6bebb0e4802d"
server_key_iv: "ece3438d6ab38a869ea3b89b3ca355da"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.989248 31744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:27.990121 31758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:27.990316 31744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:27.990432 31744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal
uuid: "bf7515477d0144fabafabe4c6aa8affa"
format_stamp: "Formatted at 2026-09-25 02:26:27 on dist-test-slave-hj7s"
server_key: "0bbdd4623dcfcc083d6d6bebb0e4802d"
server_key_iv: "ece3438d6ab38a869ea3b89b3ca355da"
server_key_version: "encryptionkey@0"
I20260925 02:26:27.990511 31744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:28.004134 31744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:28.004484 31744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:28.004649 31744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:28.005038 31744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:28.005385 31744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:28.005438 31744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.005479 31744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:28.005509 31744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.011545 31744 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.1:37763
I20260925 02:26:28.011592 31872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.1:37763 every 8 connection(s)
I20260925 02:26:28.011953 31744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/data/info.pb
I20260925 02:26:28.015097 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31744
I20260925 02:26:28.015190 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-0/wal/instance
I20260925 02:26:28.015401 11764 external_mini_cluster.cc:1428] Setting key 2197fe4817e5e622174741c19aceaa07
I20260925 02:26:28.016445 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.2:0
--local_ip_for_outbound_sockets=127.11.125.2
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--builtin_ntp_servers=127.11.125.20:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:26:28.017778 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.012698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:33607 (local address 127.11.125.62:46445)
0925 02:26:28.012763 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.012766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.013089 (+ 323us) server_negotiation.cc:407] Connection header received
0925 02:26:28.013590 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.013591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.013605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.013665 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:28.014327 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.014838 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.016126 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.016451 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.016531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.017480 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:28.017481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:28.017481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:28.017486 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:28.017504 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:28.017509 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:28.017567 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.017631 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.017664 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:26:28.017997 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.012697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46445 (local address 127.11.125.1:33607)
0925 02:26:28.012951 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.012997 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.013500 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.013746 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.013750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.013758 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:28.014255 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.014259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.014937 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.014938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.016027 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.016031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.016559 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.016561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.016730 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.017328 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:28.017339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:28.017368 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:28.017391 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:28.017573 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:28.017578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:28.017580 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.017632 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.017747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260925 02:26:28.018554 31873 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46445
I20260925 02:26:28.018657 31873 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:28.018843 31873 heartbeater.cc:511] Master 127.11.125.62:46445 requested a full tablet report, sending...
I20260925 02:26:28.019462 31684 ts_manager.cc:194] Registered new tserver with Master: bf7515477d0144fabafabe4c6aa8affa (127.11.125.1:37763)
I20260925 02:26:28.020438 31684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.1:33607
W20260925 02:26:28.100201 31876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:28.100425 31876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:28.100446 31876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:28.101640 31876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:28.101684 31876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:28.101709 31876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.2
I20260925 02:26:28.102994 31876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.125.2
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.2
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:28.103189 31876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:28.103446 31876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:28.108548 31881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:28.108539 31884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:28.108500 31882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:28.108774 31876 server_base.cc:1034] running on GCE node
I20260925 02:26:28.108999 31876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:28.109277 31876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:28.110469 31876 hybrid_clock.cc:648] HybridClock initialized: now 1790303188110464 us; error 41 us; skew 500 ppm
I20260925 02:26:28.111681 31876 webserver.cc:460] Webserver started at http://127.11.125.2:39105/ using document root <none> and password file <none>
I20260925 02:26:28.111956 31876 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:28.112026 31876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:28.112192 31876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:28.113531 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data/instance:
uuid: "8def37837f2b4d049de5b88ba048ec96"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "0d8df33eb7fb434cb85c18a6203f9b72"
server_key_iv: "1a664dab88b7ee46e1c4280e3a40f083"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.113986 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance:
uuid: "8def37837f2b4d049de5b88ba048ec96"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "0d8df33eb7fb434cb85c18a6203f9b72"
server_key_iv: "1a664dab88b7ee46e1c4280e3a40f083"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.115370 31876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:28.116055 31890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.116257 31876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:28.116357 31876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal
uuid: "8def37837f2b4d049de5b88ba048ec96"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "0d8df33eb7fb434cb85c18a6203f9b72"
server_key_iv: "1a664dab88b7ee46e1c4280e3a40f083"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.116431 31876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:28.127593 31876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:28.127905 31876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:28.128057 31876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:28.128286 31876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:28.128618 31876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:28.128650 31876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.128693 31876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:28.128720 31876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.135041 31876 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.2:42375
I20260925 02:26:28.135131 32003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.2:42375 every 8 connection(s)
I20260925 02:26:28.135456 31876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/data/info.pb
I20260925 02:26:28.140764 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.136285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.2:58325 (local address 127.11.125.62:46445)
0925 02:26:28.136330 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.136332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.136562 (+ 230us) server_negotiation.cc:407] Connection header received
0925 02:26:28.137138 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.137140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.137151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.137206 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:28.137896 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.138389 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.139378 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.139557 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.139608 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.140486 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:28.140488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:28.140488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:28.140493 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:28.140515 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:28.140519 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:28.140579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.140642 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.140675 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:26:28.141047 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.136270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46445 (local address 127.11.125.2:58325)
0925 02:26:28.136442 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.136477 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.137050 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.137261 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.137264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.137273 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:28.137814 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.137819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.138472 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.138472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.139298 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.139301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.139634 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.139635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.139743 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.140346 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:28.140347 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:28.140378 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:28.140402 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:28.140574 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:28.140578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:28.140581 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.140655 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.140757 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 02:26:28.141564 32004 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46445
I20260925 02:26:28.141667 32004 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:28.141867 32004 heartbeater.cc:511] Master 127.11.125.62:46445 requested a full tablet report, sending...
I20260925 02:26:28.142277 31684 ts_manager.cc:194] Registered new tserver with Master: 8def37837f2b4d049de5b88ba048ec96 (127.11.125.2:42375)
I20260925 02:26:28.142669 31684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.2:58325
I20260925 02:26:28.142696 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 31876
I20260925 02:26:28.142817 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal/instance
I20260925 02:26:28.143067 11764 external_mini_cluster.cc:1428] Setting key 27a7d9149dd169669276328c0a15b158
I20260925 02:26:28.144006 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.125.3:0
--local_ip_for_outbound_sockets=127.11.125.3
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--builtin_ntp_servers=127.11.125.20:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:26:28.227648 32007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:28.227859 32007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:28.227881 32007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:28.229113 32007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:28.229156 32007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:28.229182 32007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.125.3
I20260925 02:26:28.230522 32007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.125.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.125.3
--webserver_port=0
--tserver_master_addrs=127.11.125.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.125.3
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:28.230716 32007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:28.230970 32007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:28.236032 32012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:28.236037 32015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:28.236028 32013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:28.236351 32007 server_base.cc:1034] running on GCE node
I20260925 02:26:28.236575 32007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:28.236778 32007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:28.237918 32007 hybrid_clock.cc:648] HybridClock initialized: now 1790303188237912 us; error 44 us; skew 500 ppm
I20260925 02:26:28.239426 32007 webserver.cc:460] Webserver started at http://127.11.125.3:34039/ using document root <none> and password file <none>
I20260925 02:26:28.239697 32007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:28.239782 32007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:28.239945 32007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:28.241273 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data/instance:
uuid: "54abeeb8329d49d2b05ce55db74fc4d6"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "a2e919b0b29630905c165e80d22d5a23"
server_key_iv: "cf701311b0fce7b32e75bfd7fd152bab"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.241655 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance:
uuid: "54abeeb8329d49d2b05ce55db74fc4d6"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "a2e919b0b29630905c165e80d22d5a23"
server_key_iv: "cf701311b0fce7b32e75bfd7fd152bab"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.242969 32007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:28.243718 32021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.243922 32007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:28.243988 32007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal
uuid: "54abeeb8329d49d2b05ce55db74fc4d6"
format_stamp: "Formatted at 2026-09-25 02:26:28 on dist-test-slave-hj7s"
server_key: "a2e919b0b29630905c165e80d22d5a23"
server_key_iv: "cf701311b0fce7b32e75bfd7fd152bab"
server_key_version: "encryptionkey@0"
I20260925 02:26:28.244078 32007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:28.277299 32007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:28.277665 32007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:28.277834 32007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:28.278100 32007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:26:28.278482 32007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:26:28.278539 32007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.278577 32007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:26:28.278605 32007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:28.284482 32007 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.3:36219
I20260925 02:26:28.284535 32136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.3:36219 every 8 connection(s)
I20260925 02:26:28.284878 32007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/data/info.pb
I20260925 02:26:28.289448 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32007
I20260925 02:26:28.289577 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal/instance
I20260925 02:26:28.289784 11764 external_mini_cluster.cc:1428] Setting key 88c3339a98bc1aba763c74aaf8077009
I20260925 02:26:28.290400 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.285659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.3:47235 (local address 127.11.125.62:46445)
0925 02:26:28.285736 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.285752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.286016 (+ 264us) server_negotiation.cc:407] Connection header received
0925 02:26:28.286578 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.286579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.286591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.286647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:28.287381 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.287893 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.288915 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.289127 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.289207 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.290124 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:28.290126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:28.290126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:28.290131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:28.290153 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:28.290157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:28.290217 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.290291 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.290318 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:26:28.290645 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.285641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:46445 (local address 127.11.125.3:47235)
0925 02:26:28.285856 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.285906 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.286462 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.286739 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.286743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.286752 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:28.287272 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.287276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.287994 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.287996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.288783 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.288785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.289218 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.289219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.289371 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.289973 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:26:28.289973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:28.290003 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:26:28.290026 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:28.290219 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:28.290225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:28.290228 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.290297 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.290410 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260925 02:26:28.291115 32137 heartbeater.cc:347] Connected to a master server at 127.11.125.62:46445
I20260925 02:26:28.291198 32137 heartbeater.cc:464] Registering TS with master...
I20260925 02:26:28.291400 32137 heartbeater.cc:511] Master 127.11.125.62:46445 requested a full tablet report, sending...
I20260925 02:26:28.291894 31684 ts_manager.cc:194] Registered new tserver with Master: 54abeeb8329d49d2b05ce55db74fc4d6 (127.11.125.3:36219)
I20260925 02:26:28.292258 31684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.125.3:47235
I20260925 02:26:28.301745 11764 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:26:28.305872 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.303187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.11.125.62:46445)
0925 02:26:28.303246 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.303249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.303401 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:26:28.303442 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.303443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.303452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.303498 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:28.303624 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.304100 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.304790 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.305039 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.305087 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.305636 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:28.305637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:28.305638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:28.305642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:28.305655 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:28.305659 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:28.305699 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.305732 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.305751 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 02:26:28.307358 31684 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:26:28.308982 31684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:26:28.315336 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.11.125.3:36219)
0925 02:26:28.311730 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.311735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.311787 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:26:28.311937 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.311939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.311953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.312022 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:28.312196 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.312886 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.314271 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.314927 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.315017 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.315117 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.315206 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.315247 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260925 02:26:28.315323 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:36219 (local address 127.0.0.1:33466)
0925 02:26:28.311731 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.311740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.311795 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.312083 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.312084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.312086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:28.312122 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.312123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.313016 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.313017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.314176 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.314178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.315020 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.315021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.315101 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.315114 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.315191 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.315247 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":77,"threads_started":1}
I20260925 02:26:28.315773 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:42375 (local address 127.0.0.1:51692)
0925 02:26:28.311572 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.311615 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.311688 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.312046 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.312047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.312049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:28.312128 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.312130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.313458 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.313459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.314483 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.314485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.315531 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.315532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.315598 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.315602 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.315657 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.315695 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":28,"thread_start_us":57,"threads_started":1}
I20260925 02:26:28.316057 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37763 (local address 127.0.0.1:52526)
0925 02:26:28.311575 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.311616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.311688 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.312021 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.312024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.312026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:28.312092 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.312094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.313482 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.313483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.314541 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.314544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.315829 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.315830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.315901 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.315904 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.315940 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.315993 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
I20260925 02:26:28.316195 32069 tablet_service.cc:1467] Processing CreateTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 (DEFAULT_TABLE table=test-table [id=56dde3fa1eb249fcb43f08d11db6619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:28.316491 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.11.125.1:37763)
0925 02:26:28.311579 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.311583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.311773 (+ 190us) server_negotiation.cc:407] Connection header received
0925 02:26:28.311824 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.311827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.311855 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.311929 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:28.312170 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.313383 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.314615 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.315312 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.316198 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.316310 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.316349 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.316399 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260925 02:26:28.316624 32069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2280148aab44ac5b5ccfaeef19bcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:28.317126 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.311339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.11.125.2:42375)
0925 02:26:28.311664 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.311668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.311679 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:28.311878 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.311881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.311898 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.311971 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:28.312213 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.313361 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.314563 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.315311 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.315386 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.315487 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.316987 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.317044 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260925 02:26:28.317128 31807 tablet_service.cc:1467] Processing CreateTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 (DEFAULT_TABLE table=test-table [id=56dde3fa1eb249fcb43f08d11db6619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:28.317446 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2280148aab44ac5b5ccfaeef19bcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:28.317909 31938 tablet_service.cc:1467] Processing CreateTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 (DEFAULT_TABLE table=test-table [id=56dde3fa1eb249fcb43f08d11db6619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:26:28.318272 31938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2280148aab44ac5b5ccfaeef19bcc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:28.320155 32151 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Bootstrap starting.
I20260925 02:26:28.320889 32151 tablet_bootstrap.cc:654] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:28.320953 32152 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: Bootstrap starting.
I20260925 02:26:28.321386 32151 log.cc:824] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:28.321609 32152 tablet_bootstrap.cc:654] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:28.321640 32153 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Bootstrap starting.
I20260925 02:26:28.321957 32152 log.cc:824] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:28.322269 32153 tablet_bootstrap.cc:654] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:28.322324 32151 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: No bootstrap required, opened a new log
I20260925 02:26:28.322386 32151 ts_tablet_manager.cc:1397] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:28.322616 32153 log.cc:824] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:28.322929 32152 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: No bootstrap required, opened a new log
I20260925 02:26:28.323007 32152 ts_tablet_manager.cc:1397] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:28.323316 32153 tablet_bootstrap.cc:492] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: No bootstrap required, opened a new log
I20260925 02:26:28.323406 32153 ts_tablet_manager.cc:1397] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:26:28.325095 32151 raft_consensus.cc:348] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.325152 32153 raft_consensus.cc:348] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.325278 32151 raft_consensus.cc:374] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:28.325296 32153 raft_consensus.cc:374] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:28.325315 32151 raft_consensus.cc:729] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54abeeb8329d49d2b05ce55db74fc4d6, State: Initialized, Role: FOLLOWER
I20260925 02:26:28.325350 32153 raft_consensus.cc:729] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8def37837f2b4d049de5b88ba048ec96, State: Initialized, Role: FOLLOWER
I20260925 02:26:28.325459 32153 consensus_queue.cc:260] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.325452 32151 consensus_queue.cc:260] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.325749 32151 ts_tablet_manager.cc:1428] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 02:26:28.325752 32153 ts_tablet_manager.cc:1428] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 02:26:28.325784 32004 heartbeater.cc:503] Master 127.11.125.62:46445 was elected leader, sending a full tablet report...
I20260925 02:26:28.325935 32137 heartbeater.cc:503] Master 127.11.125.62:46445 was elected leader, sending a full tablet report...
I20260925 02:26:28.326043 32152 raft_consensus.cc:348] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.326221 32152 raft_consensus.cc:374] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:28.326262 32152 raft_consensus.cc:729] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7515477d0144fabafabe4c6aa8affa, State: Initialized, Role: FOLLOWER
I20260925 02:26:28.326367 32152 consensus_queue.cc:260] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.326622 32152 ts_tablet_manager.cc:1428] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 02:26:28.326674 31873 heartbeater.cc:503] Master 127.11.125.62:46445 was elected leader, sending a full tablet report...
W20260925 02:26:28.386127 32005 tablet.cc:2369] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:26:28.407886 32159 raft_consensus.cc:482] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:26:28.408000 32159 raft_consensus.cc:504] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.408310 32159 leader_election.cc:288] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8def37837f2b4d049de5b88ba048ec96 (127.11.125.2:42375), 54abeeb8329d49d2b05ce55db74fc4d6 (127.11.125.3:36219)
I20260925 02:26:28.412312 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.408695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:45435 (local address 127.11.125.3:36219)
0925 02:26:28.408743 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.408746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.408838 (+ 92us) server_negotiation.cc:407] Connection header received
0925 02:26:28.408930 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.408931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.408940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.408997 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:28.409124 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.409795 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.411320 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.411905 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.411991 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.412002 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.412185 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.412225 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:26:28.412298 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.408626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.125.1:38327 (local address 127.11.125.2:42375)
0925 02:26:28.408678 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.408681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.408693 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:26:28.408724 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.408724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.408736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.408789 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:26:28.408993 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.409771 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.411320 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.411961 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.412031 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.412038 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.412190 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.412227 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:26:28.412343 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.408509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.2:42375 (local address 127.11.125.1:38327)
0925 02:26:28.408581 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.408595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.408660 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.408808 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.408809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.408810 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:28.408883 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.408885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.409877 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.409878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.411222 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.411225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.412041 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.412041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.412111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.412116 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.412172 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.412218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":86}
I20260925 02:26:28.412341 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.408618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.3:36219 (local address 127.11.125.1:45435)
0925 02:26:28.408776 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.408784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:28.408834 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:28.409005 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:28.409005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:28.409007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:26:28.409064 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.409065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.409877 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.409878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.411222 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:28.411225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.412029 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.412031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:28.412116 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.412119 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:26:28.412172 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:26:28.412218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":59,"thread_start_us":106,"threads_started":1}
I20260925 02:26:28.412767 31958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2280148aab44ac5b5ccfaeef19bcc14" candidate_uuid: "bf7515477d0144fabafabe4c6aa8affa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8def37837f2b4d049de5b88ba048ec96" is_pre_election: true
I20260925 02:26:28.412989 31958 raft_consensus.cc:2454] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7515477d0144fabafabe4c6aa8affa in term 0.
I20260925 02:26:28.412994 32090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2280148aab44ac5b5ccfaeef19bcc14" candidate_uuid: "bf7515477d0144fabafabe4c6aa8affa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" is_pre_election: true
I20260925 02:26:28.413174 32090 raft_consensus.cc:2454] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7515477d0144fabafabe4c6aa8affa in term 0.
I20260925 02:26:28.413250 31759 leader_election.cc:302] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8def37837f2b4d049de5b88ba048ec96, bf7515477d0144fabafabe4c6aa8affa; no voters:
I20260925 02:26:28.413398 32159 raft_consensus.cc:2781] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:26:28.413458 32159 raft_consensus.cc:482] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:26:28.413484 32159 raft_consensus.cc:3037] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:28.414182 32159 raft_consensus.cc:504] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.414317 32159 leader_election.cc:288] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [CANDIDATE]: Term 1 election: Requested vote from peers 8def37837f2b4d049de5b88ba048ec96 (127.11.125.2:42375), 54abeeb8329d49d2b05ce55db74fc4d6 (127.11.125.3:36219)
I20260925 02:26:28.414467 31958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2280148aab44ac5b5ccfaeef19bcc14" candidate_uuid: "bf7515477d0144fabafabe4c6aa8affa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8def37837f2b4d049de5b88ba048ec96"
I20260925 02:26:28.414510 32090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2280148aab44ac5b5ccfaeef19bcc14" candidate_uuid: "bf7515477d0144fabafabe4c6aa8affa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54abeeb8329d49d2b05ce55db74fc4d6"
I20260925 02:26:28.414567 31958 raft_consensus.cc:3037] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:28.414601 32090 raft_consensus.cc:3037] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:28.415290 31958 raft_consensus.cc:2454] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7515477d0144fabafabe4c6aa8affa in term 1.
I20260925 02:26:28.415403 32090 raft_consensus.cc:2454] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7515477d0144fabafabe4c6aa8affa in term 1.
I20260925 02:26:28.415472 31759 leader_election.cc:302] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8def37837f2b4d049de5b88ba048ec96, bf7515477d0144fabafabe4c6aa8affa; no voters:
I20260925 02:26:28.415571 32159 raft_consensus.cc:2781] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:28.415763 32159 raft_consensus.cc:686] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [term 1 LEADER]: Becoming Leader. State: Replica: bf7515477d0144fabafabe4c6aa8affa, State: Running, Role: LEADER
I20260925 02:26:28.415902 32159 consensus_queue.cc:237] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } }
I20260925 02:26:28.416833 31682 catalog_manager.cc:5495] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7515477d0144fabafabe4c6aa8affa (127.11.125.1). New cstate: current_term: 1 leader_uuid: "bf7515477d0144fabafabe4c6aa8affa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7515477d0144fabafabe4c6aa8affa" member_type: VOTER last_known_addr { host: "127.11.125.1" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260925 02:26:28.432456 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.430076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.11.125.1:37763)
0925 02:26:28.430141 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:28.430143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:28.430153 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:28.430197 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:28.430198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:28.430225 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:28.430300 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:28.430420 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.430884 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.431657 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:28.431850 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:28.431925 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:28.432097 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:28.432196 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:28.432290 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:28.432337 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:26:28.432355 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:26:28.434330 32090 raft_consensus.cc:1261] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 1 FOLLOWER]: Refusing update from remote peer bf7515477d0144fabafabe4c6aa8affa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:28.434345 31958 raft_consensus.cc:1261] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 1 FOLLOWER]: Refusing update from remote peer bf7515477d0144fabafabe4c6aa8affa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:28.434691 32159 consensus_queue.cc:1035] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [LEADER]: Connected to new peer: Peer: permanent_uuid: "54abeeb8329d49d2b05ce55db74fc4d6" member_type: VOTER last_known_addr { host: "127.11.125.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:28.434860 32161 consensus_queue.cc:1035] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8def37837f2b4d049de5b88ba048ec96" member_type: VOTER last_known_addr { host: "127.11.125.2" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:28.437644 32165 mvcc.cc:205] Tried to move back new op lower bound from 7333081859824304128 to 7333081859752239104. Current Snapshot: MvccSnapshot[applied={T|T < 7333081859824304128}]
I20260925 02:26:28.439687 31682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:28.439795 31682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:26:28.440603 31682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65cd744e7e3a48dda5419396664d89e1: Sending DeleteTablet for 3 replicas of tablet d2280148aab44ac5b5ccfaeef19bcc14
I20260925 02:26:28.440979 31807 tablet_service.cc:1514] Processing DeleteTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:28 UTC) from {username='slave'} at 127.0.0.1:52526
I20260925 02:26:28.441195 32069 tablet_service.cc:1514] Processing DeleteTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:28 UTC) from {username='slave'} at 127.0.0.1:33466
I20260925 02:26:28.441201 31938 tablet_service.cc:1514] Processing DeleteTablet for tablet d2280148aab44ac5b5ccfaeef19bcc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:26:28 UTC) from {username='slave'} at 127.0.0.1:51692
I20260925 02:26:28.441468 32175 tablet_replica.cc:330] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: stopping tablet replica
I20260925 02:26:28.441577 32175 raft_consensus.cc:2229] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:28.441685 32175 raft_consensus.cc:2258] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:28.441824 32174 tablet_replica.cc:330] T d2280148aab44ac5b5ccfaeef19bcc14 P bf7515477d0144fabafabe4c6aa8affa: stopping tablet replica
I20260925 02:26:28.441857 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31744
I20260925 02:26:28.443249 32175 ts_tablet_manager.cc:1905] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:28.444514 32176 tablet_replica.cc:330] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: stopping tablet replica
I20260925 02:26:28.444653 32176 raft_consensus.cc:2229] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:26:28.444764 32176 raft_consensus.cc:2258] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:26:28.444978 32175 ts_tablet_manager.cc:1918] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:28.445055 32175 log.cc:1196] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-2/wal/wals/d2280148aab44ac5b5ccfaeef19bcc14
I20260925 02:26:28.445297 32175 ts_tablet_manager.cc:1939] T d2280148aab44ac5b5ccfaeef19bcc14 P 54abeeb8329d49d2b05ce55db74fc4d6: Deleting consensus metadata
I20260925 02:26:28.445765 31668 catalog_manager.cc:4841] TS 54abeeb8329d49d2b05ce55db74fc4d6 (127.11.125.3:36219): tablet d2280148aab44ac5b5ccfaeef19bcc14 (table test-table [id=56dde3fa1eb249fcb43f08d11db6619a]) successfully deleted
I20260925 02:26:28.446035 32176 ts_tablet_manager.cc:1905] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:26:28.447412 32176 ts_tablet_manager.cc:1918] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:26:28.447474 32176 log.cc:1196] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Deleting WAL directory at /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790302942477976-11764-0/minicluster-data/ts-1/wal/wals/d2280148aab44ac5b5ccfaeef19bcc14
I20260925 02:26:28.447702 32176 ts_tablet_manager.cc:1939] T d2280148aab44ac5b5ccfaeef19bcc14 P 8def37837f2b4d049de5b88ba048ec96: Deleting consensus metadata
I20260925 02:26:28.448273 31668 catalog_manager.cc:4841] TS 8def37837f2b4d049de5b88ba048ec96 (127.11.125.2:42375): tablet d2280148aab44ac5b5ccfaeef19bcc14 (table test-table [id=56dde3fa1eb249fcb43f08d11db6619a]) successfully deleted
W20260925 02:26:28.449023 31670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.125.1:37763 (error 108)
I20260925 02:26:28.449239 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31876
I20260925 02:26:28.449658 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:28.449530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.1:37763 (local address 127.0.0.1:52540)
0925 02:26:28.449577 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:28.449619 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 02:26:28.449764 31670 catalog_manager.cc:4584] TS bf7515477d0144fabafabe4c6aa8affa (127.11.125.1:37763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2280148aab44ac5b5ccfaeef19bcc14: Network error: Client connection negotiation failed: client connection to 127.11.125.1:37763: connect: Connection refused (error 111)
I20260925 02:26:28.455260 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32007
I20260925 02:26:28.459971 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 31653
2026-09-25T02:26:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (769 ms)
[----------] 6 tests from EncryptionPolicyTest (9340 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:28 dist-test-slave-hj7s krb5kdc[32180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:28 dist-test-slave-hj7s krb5kdc[32180](info): set up 2 sockets
Sep 25 02:26:28 dist-test-slave-hj7s krb5kdc[32180](info): commencing operation
krb5kdc: starting...
W20260925 02:26:29.507122 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:29.627298 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:29.710665 32194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:29.710925 32194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:29.710966 32194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:29.712229 32194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:29.712303 32194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:29.712327 32194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:29.712348 32194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:29.712378 32194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:29.713913 32194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:29.714195 32194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:29.714494 32194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:29.719806 32199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:29.719828 32194 server_base.cc:1034] running on GCE node
W20260925 02:26:29.719796 32200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:29.719796 32202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:29.720356 32194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:29.720602 32194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:29.721736 32194 hybrid_clock.cc:648] HybridClock initialized: now 1790303189721722 us; error 30 us; skew 500 ppm
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:29.730880 32194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:29.731396 32194 webserver.cc:460] Webserver started at http://127.0.0.1:41701/ using document root <none> and password file <none>
I20260925 02:26:29.731633 32194 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:29.731679 32194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:29.731837 32194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:29.732999 32194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "7db73c8ce2c047dab7cece0fd8d30609"
format_stamp: "Formatted at 2026-09-25 02:26:29 on dist-test-slave-hj7s"
server_key: "548ae0c40b940d3ca060ce3e7d8a894b"
server_key_iv: "d2d51dae5b6e2db29ba862a115ee5b93"
server_key_version: "encryptionkey@0"
I20260925 02:26:29.733389 32194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "7db73c8ce2c047dab7cece0fd8d30609"
format_stamp: "Formatted at 2026-09-25 02:26:29 on dist-test-slave-hj7s"
server_key: "548ae0c40b940d3ca060ce3e7d8a894b"
server_key_iv: "d2d51dae5b6e2db29ba862a115ee5b93"
server_key_version: "encryptionkey@0"
I20260925 02:26:29.734727 32194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 02:26:29.735462 32209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:29.735637 32194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 02:26:29.735747 32194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "7db73c8ce2c047dab7cece0fd8d30609"
format_stamp: "Formatted at 2026-09-25 02:26:29 on dist-test-slave-hj7s"
server_key: "548ae0c40b940d3ca060ce3e7d8a894b"
server_key_iv: "d2d51dae5b6e2db29ba862a115ee5b93"
server_key_version: "encryptionkey@0"
I20260925 02:26:29.735821 32194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:29.753216 32194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:29.755056 32194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:29.755221 32194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:29.759516 32194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32985
I20260925 02:26:29.759558 32261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32985 every 8 connection(s)
I20260925 02:26:29.759945 32194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:29.760797 32262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:29.762020 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32194
I20260925 02:26:29.762151 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:29.762346 11764 external_mini_cluster.cc:1428] Setting key 7ea0caee21be27168a4ae41457a0a361
I20260925 02:26:29.763662 32262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609: Bootstrap starting.
I20260925 02:26:29.764312 32262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:29.764609 32262 log.cc:824] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:29.765439 32262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609: No bootstrap required, opened a new log
I20260925 02:26:29.767725 32262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260925 02:26:29.767913 32262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:29.767954 32262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db73c8ce2c047dab7cece0fd8d30609, State: Initialized, Role: FOLLOWER
I20260925 02:26:29.768102 32262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260925 02:26:29.768190 32262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:29.768214 32262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:29.768285 32262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:29.768868 32262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
I20260925 02:26:29.769017 32262 leader_election.cc:302] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db73c8ce2c047dab7cece0fd8d30609; no voters:
I20260925 02:26:29.769205 32262 leader_election.cc:288] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:29.769268 32268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:29.769431 32268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [term 1 LEADER]: Becoming Leader. State: Replica: 7db73c8ce2c047dab7cece0fd8d30609, State: Running, Role: LEADER
I20260925 02:26:29.769474 32262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:29.769541 32268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } }
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:26:29.769887 32268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db73c8ce2c047dab7cece0fd8d30609. Latest consensus state: current_term: 1 leader_uuid: "7db73c8ce2c047dab7cece0fd8d30609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260925 02:26:29.769953 32268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:29.770362 32275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:29.770885 32275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:29.771049 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db73c8ce2c047dab7cece0fd8d30609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db73c8ce2c047dab7cece0fd8d30609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32985 } } }
I20260925 02:26:29.771273 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:29.773373 32275 catalog_manager.cc:1347] Generated new cluster ID: 83b1fdd498684bf2a6db6c2fbabb531f
I20260925 02:26:29.773451 32275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:29.774854 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:29.762909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58583 (local address 127.0.0.1:32985)
0925 02:26:29.763115 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:29.763127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:29.763163 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:26:29.763704 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:29.763713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:29.763773 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:29.763916 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:29.764640 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:29.766084 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:29.766828 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:29.767525 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:29.767641 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:29.770494 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:29.770498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:29.770499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:29.770511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:29.772645 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:29.773104 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:29.773107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:29.773108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:29.773230 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:29.773543 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:29.773544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:29.773544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:29.773839 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:29.774448 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:29.774542 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 02:26:29.774645 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260925 02:26:29.775853 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:29.784837 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:29.777314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44741 (local address 127.0.0.1:32985)
0925 02:26:29.777347 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:29.777348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:29.777482 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:26:29.777525 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:29.777526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:29.777535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:29.777590 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:29.778200 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:29.779232 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:29.780275 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:29.780602 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:29.780659 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:29.782041 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:29.782042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:29.782042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:29.782049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:29.783569 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:29.783986 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:29.783987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:29.783987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:29.784041 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:29.784267 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:29.784268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:29.784268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:29.784380 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:29.784480 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:29.784689 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:26:29.784747 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:26:29.803663 32275 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:29.804206 32275 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:29.806754 32275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db73c8ce2c047dab7cece0fd8d30609: Generated new TSK 0
I20260925 02:26:29.806958 32275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:29.810266 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32194
W20260925 02:26:29.814957 32178 connection.cc:535] client connection to 127.0.0.1:32985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32985): Connection reset by peer (error 104)
2026-09-25T02:26:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32294](info): set up 2 sockets
Sep 25 02:26:29 dist-test-slave-hj7s krb5kdc[32294](info): commencing operation
krb5kdc: starting...
W20260925 02:26:30.863121 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:30 dist-test-slave-hj7s krb5kdc[32294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:30.984407 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:31.076202 32307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:31.076489 32307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:31.076529 32307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:31.077787 32307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:31.077857 32307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:31.077883 32307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:31.077909 32307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:31.077930 32307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:31.079833 32307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:31.080118 32307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:31.080386 32307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:31.086614 32313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:31.086601 32312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:31.086599 32315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:31.087028 32307 server_base.cc:1034] running on GCE node
I20260925 02:26:31.087298 32307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:31.087600 32307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:31.088764 32307 hybrid_clock.cc:648] HybridClock initialized: now 1790303191088726 us; error 32 us; skew 500 ppm
Sep 25 02:26:31 dist-test-slave-hj7s krb5kdc[32294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:31.092196 32307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:31.092756 32307 webserver.cc:460] Webserver started at http://127.0.0.1:37921/ using document root <none> and password file <none>
I20260925 02:26:31.093055 32307 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:31.093132 32307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:31.093293 32307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:31.094781 32307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "16fc143bb1cd4d8893e70205aa025f9b"
format_stamp: "Formatted at 2026-09-25 02:26:31 on dist-test-slave-hj7s"
server_key: "9a5fd53e331ad60dbf962c6effa42808"
server_key_iv: "e04a961e9eece353717d42970875ee99"
server_key_version: "encryptionkey@0"
I20260925 02:26:31.095234 32307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "16fc143bb1cd4d8893e70205aa025f9b"
format_stamp: "Formatted at 2026-09-25 02:26:31 on dist-test-slave-hj7s"
server_key: "9a5fd53e331ad60dbf962c6effa42808"
server_key_iv: "e04a961e9eece353717d42970875ee99"
server_key_version: "encryptionkey@0"
I20260925 02:26:31.097189 32307 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:26:31.098183 32322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:31.098405 32307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:31.098511 32307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "16fc143bb1cd4d8893e70205aa025f9b"
format_stamp: "Formatted at 2026-09-25 02:26:31 on dist-test-slave-hj7s"
server_key: "9a5fd53e331ad60dbf962c6effa42808"
server_key_iv: "e04a961e9eece353717d42970875ee99"
server_key_version: "encryptionkey@0"
I20260925 02:26:31.098591 32307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:31.111747 32307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:31.113636 32307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:31.113797 32307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:31.118247 32307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41299
I20260925 02:26:31.118278 32374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41299 every 8 connection(s)
I20260925 02:26:31.118662 32307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:31.119527 32375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:31.119592 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32307
I20260925 02:26:31.119733 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:31.119956 11764 external_mini_cluster.cc:1428] Setting key b075ff141930fc2795bc0644d58e0222
I20260925 02:26:31.122918 32375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b: Bootstrap starting.
I20260925 02:26:31.123639 32375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:31.123940 32375 log.cc:824] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:31.124665 32375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b: No bootstrap required, opened a new log
I20260925 02:26:31.125643 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:31.120494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44325 (local address 127.0.0.1:41299)
0925 02:26:31.120675 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:31.120683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:31.120709 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:26:31.121227 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:31.121234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:31.121282 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:31.121396 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:31.121627 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:31.122473 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:31.123587 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:31.124053 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:31.124146 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:31.124958 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:31.124963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:31.124963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:31.125157 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:31.125196 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:31.125210 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:31.125303 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:31.125398 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:26:31.125425 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260925 02:26:31.126751 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:31.127228 32375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260925 02:26:31.127368 32375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:31.127391 32375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16fc143bb1cd4d8893e70205aa025f9b, State: Initialized, Role: FOLLOWER
I20260925 02:26:31.127526 32375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260925 02:26:31.127595 32375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:31.127617 32375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:31.127679 32375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:31.128329 32375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260925 02:26:31.128443 32375 leader_election.cc:302] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16fc143bb1cd4d8893e70205aa025f9b; no voters:
I20260925 02:26:31.128626 32375 leader_election.cc:288] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:31.128655 32385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:31.128790 32385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [term 1 LEADER]: Becoming Leader. State: Replica: 16fc143bb1cd4d8893e70205aa025f9b, State: Running, Role: LEADER
I20260925 02:26:31.128949 32375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:31.128934 32385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260925 02:26:31.129311 32386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16fc143bb1cd4d8893e70205aa025f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } } }
I20260925 02:26:31.129398 32386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:31.129631 32387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16fc143bb1cd4d8893e70205aa025f9b. Latest consensus state: current_term: 1 leader_uuid: "16fc143bb1cd4d8893e70205aa025f9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16fc143bb1cd4d8893e70205aa025f9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } } }
I20260925 02:26:31.129765 32387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:31.129923 32391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:31.130461 32391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:31.131729 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:31.128185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57721 (local address 127.0.0.1:41299)
0925 02:26:31.128223 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:31.128225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:31.128300 (+ 75us) server_negotiation.cc:407] Connection header received
0925 02:26:31.128347 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:31.128347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:31.128357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:31.128416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:31.128599 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:31.129424 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:31.130433 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:31.130705 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:31.131056 (+ 351us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:31.131481 (+ 425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:31.131482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:31.131483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:31.131487 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:31.131501 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:31.131506 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:31.131574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:31.131641 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:26:31.131652 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 02:26:31.132090 32391 catalog_manager.cc:1347] Generated new cluster ID: 50f5841c57674ebdba2e623d91e7ccff
I20260925 02:26:31.132144 32391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:31.145143 32391 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:31.145818 32391 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:31.156664 32391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16fc143bb1cd4d8893e70205aa025f9b: Generated new TSK 0
I20260925 02:26:31.156867 32391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:31.166694 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32307
W20260925 02:26:31.171762 32292 connection.cc:535] client connection to 127.0.0.1:41299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41299): Connection reset by peer (error 104)
2026-09-25T02:26:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:31 dist-test-slave-hj7s krb5kdc[32407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:31 dist-test-slave-hj7s krb5kdc[32407](info): set up 2 sockets
Sep 25 02:26:31 dist-test-slave-hj7s krb5kdc[32407](info): commencing operation
krb5kdc: starting...
W20260925 02:26:32.223037 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:32 dist-test-slave-hj7s krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:32.341526 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:32.422514 32420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:32.422713 32420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:32.422736 32420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:32.423805 32420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:32.423839 32420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:32.423853 32420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:32.423866 32420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:32.423878 32420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:32.425293 32420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:32.425490 32420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:32.425657 32420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:32.430617 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:32.430624 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:32.430650 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:32.430651 32420 server_base.cc:1034] running on GCE node
I20260925 02:26:32.431159 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:32.431478 32420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:32.432655 32420 hybrid_clock.cc:648] HybridClock initialized: now 1790303192432615 us; error 60 us; skew 500 ppm
Sep 25 02:26:32 dist-test-slave-hj7s krb5kdc[32407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:32.435228 32420 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:32.435730 32420 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
I20260925 02:26:32.435961 32420 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:32.436030 32420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:32.436187 32420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:32.437337 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "d584d75330974ff9bab1b44f51f5cab3"
format_stamp: "Formatted at 2026-09-25 02:26:32 on dist-test-slave-hj7s"
server_key: "10043a55790adf6b9789a479f907b2b3"
server_key_iv: "ce4ce68925ca73267539104cb3fd7e49"
server_key_version: "encryptionkey@0"
I20260925 02:26:32.437718 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "d584d75330974ff9bab1b44f51f5cab3"
format_stamp: "Formatted at 2026-09-25 02:26:32 on dist-test-slave-hj7s"
server_key: "10043a55790adf6b9789a479f907b2b3"
server_key_iv: "ce4ce68925ca73267539104cb3fd7e49"
server_key_version: "encryptionkey@0"
I20260925 02:26:32.439074 32420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:32.439790 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:32.439967 32420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:32.440079 32420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "d584d75330974ff9bab1b44f51f5cab3"
format_stamp: "Formatted at 2026-09-25 02:26:32 on dist-test-slave-hj7s"
server_key: "10043a55790adf6b9789a479f907b2b3"
server_key_iv: "ce4ce68925ca73267539104cb3fd7e49"
server_key_version: "encryptionkey@0"
I20260925 02:26:32.440172 32420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:32.443604 32420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:32.445546 32420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:32.445708 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:32.449914 32420 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35711
I20260925 02:26:32.449944 32487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35711 every 8 connection(s)
I20260925 02:26:32.450280 32420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:32.451289 32488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:32.453675 32488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3: Bootstrap starting.
I20260925 02:26:32.454317 32488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:32.454607 32488 log.cc:824] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:32.455299 32488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3: No bootstrap required, opened a new log
I20260925 02:26:32.456549 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32420
I20260925 02:26:32.456688 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:32.456895 11764 external_mini_cluster.cc:1428] Setting key 3a2e107f5320f541bda38e53d32d9899
I20260925 02:26:32.457379 32488 raft_consensus.cc:348] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260925 02:26:32.457578 32488 raft_consensus.cc:374] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:32.457670 32488 raft_consensus.cc:729] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584d75330974ff9bab1b44f51f5cab3, State: Initialized, Role: FOLLOWER
I20260925 02:26:32.457837 32488 consensus_queue.cc:260] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260925 02:26:32.457916 32488 raft_consensus.cc:388] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:32.457937 32488 raft_consensus.cc:482] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:32.458005 32488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:32.459038 32488 raft_consensus.cc:504] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260925 02:26:32.459167 32488 leader_election.cc:302] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d584d75330974ff9bab1b44f51f5cab3; no voters:
I20260925 02:26:32.459367 32488 leader_election.cc:288] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:32.459393 32493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:32.459463 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:32.457544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47943 (local address 127.0.0.1:35711)
0925 02:26:32.457766 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:32.457776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:32.457813 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:26:32.457968 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:32.457975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:32.458007 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:32.458138 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:32.458873 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:32.458877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:32.458877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:32.458882 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:32.458929 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:32.458940 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:32.459017 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:32.459087 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:26:32.459216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 02:26:32.459506 32493 raft_consensus.cc:686] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [term 1 LEADER]: Becoming Leader. State: Replica: d584d75330974ff9bab1b44f51f5cab3, State: Running, Role: LEADER
I20260925 02:26:32.459777 32493 consensus_queue.cc:237] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260925 02:26:32.459636 32488 sys_catalog.cc:564] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:32.460384 32495 sys_catalog.cc:455] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d584d75330974ff9bab1b44f51f5cab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260925 02:26:32.460569 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:32.460764 32494 sys_catalog.cc:455] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d584d75330974ff9bab1b44f51f5cab3. Latest consensus state: current_term: 1 leader_uuid: "d584d75330974ff9bab1b44f51f5cab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d584d75330974ff9bab1b44f51f5cab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260925 02:26:32.460986 32494 sys_catalog.cc:458] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:32.460870 32495 sys_catalog.cc:458] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:32.461391 32502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:32.462014 32502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:32.463706 32502 catalog_manager.cc:1347] Generated new cluster ID: ffc729db2eb74e998fca1ad905f6391a
I20260925 02:26:32.463765 32502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:32.465030 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:32.462418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37093 (local address 127.0.0.1:35711)
0925 02:26:32.462457 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:32.462459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:32.462657 (+ 198us) server_negotiation.cc:407] Connection header received
0925 02:26:32.462712 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:32.462713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:32.462718 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:32.462781 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:32.463554 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:32.463555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:32.463555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:32.463559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:32.463576 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:32.463581 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:32.463639 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:32.463741 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:26:32.463757 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:32.480528 32502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:32.480751 32502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:32.485150 32502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d584d75330974ff9bab1b44f51f5cab3: Generated new TSK 0
I20260925 02:26:32.485373 32502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:32.491230 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32420
2026-09-25T02:26:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:32 dist-test-slave-hj7s krb5kdc[32519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:32 dist-test-slave-hj7s krb5kdc[32519](info): set up 2 sockets
Sep 25 02:26:32 dist-test-slave-hj7s krb5kdc[32519](info): commencing operation
krb5kdc: starting...
W20260925 02:26:33.568300 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:33.689073 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:33.771098 32532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:33.771366 32532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:33.771409 32532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:33.772550 32532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:33.772624 32532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:33.772648 32532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:33.772675 32532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:33.772701 32532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:33.774219 32532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:33.774497 32532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:33.774722 32532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:33.779645 32538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:33.779634 32537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:33.779672 32532 server_base.cc:1034] running on GCE node
W20260925 02:26:33.779636 32540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:33.780058 32532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:33.780364 32532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:33.781523 32532 hybrid_clock.cc:648] HybridClock initialized: now 1790303193781499 us; error 47 us; skew 500 ppm
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:33.784075 32532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:33.784519 32532 webserver.cc:460] Webserver started at http://127.0.0.1:41957/ using document root <none> and password file <none>
I20260925 02:26:33.784745 32532 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:33.784813 32532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:33.784994 32532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:33.786078 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "7a06e7d4074a4c7aa41c0f145a582cff"
format_stamp: "Formatted at 2026-09-25 02:26:33 on dist-test-slave-hj7s"
server_key: "c660b6a1f3901cea94c5c78c189732f4"
server_key_iv: "339d3878d72bcfc87588e4b7cf233d69"
server_key_version: "encryptionkey@0"
I20260925 02:26:33.786494 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "7a06e7d4074a4c7aa41c0f145a582cff"
format_stamp: "Formatted at 2026-09-25 02:26:33 on dist-test-slave-hj7s"
server_key: "c660b6a1f3901cea94c5c78c189732f4"
server_key_iv: "339d3878d72bcfc87588e4b7cf233d69"
server_key_version: "encryptionkey@0"
I20260925 02:26:33.787947 32532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:33.788842 32548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:33.789074 32532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:26:33.789211 32532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "7a06e7d4074a4c7aa41c0f145a582cff"
format_stamp: "Formatted at 2026-09-25 02:26:33 on dist-test-slave-hj7s"
server_key: "c660b6a1f3901cea94c5c78c189732f4"
server_key_iv: "339d3878d72bcfc87588e4b7cf233d69"
server_key_version: "encryptionkey@0"
I20260925 02:26:33.789295 32532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:33.804847 32532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:33.807420 32532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:33.807587 32532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:33.811779 32532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37421
I20260925 02:26:33.811812 32600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37421 every 8 connection(s)
I20260925 02:26:33.812183 32532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:33.813153 32601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:33.813577 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32532
I20260925 02:26:33.813709 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:33.813892 11764 external_mini_cluster.cc:1428] Setting key ec4a9c8bd9ba36c0beefeda632bd18de
I20260925 02:26:33.816340 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff: Bootstrap starting.
I20260925 02:26:33.817071 32601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:33.817365 32601 log.cc:824] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:33.818109 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff: No bootstrap required, opened a new log
I20260925 02:26:33.820245 32601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260925 02:26:33.820411 32601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:33.820448 32601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a06e7d4074a4c7aa41c0f145a582cff, State: Initialized, Role: FOLLOWER
I20260925 02:26:33.820575 32601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260925 02:26:33.820659 32601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:33.820690 32601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:33.820744 32601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:26:33.821509 32601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260925 02:26:33.821628 32601 leader_election.cc:302] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a06e7d4074a4c7aa41c0f145a582cff; no voters:
I20260925 02:26:33.821837 32601 leader_election.cc:288] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:33.821857 32606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:33.821983 32606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [term 1 LEADER]: Becoming Leader. State: Replica: 7a06e7d4074a4c7aa41c0f145a582cff, State: Running, Role: LEADER
I20260925 02:26:33.822120 32601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:33.822122 32606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } }
I20260925 02:26:33.822835 32608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } } }
I20260925 02:26:33.823072 32608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:33.823004 32607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a06e7d4074a4c7aa41c0f145a582cff. Latest consensus state: current_term: 1 leader_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a06e7d4074a4c7aa41c0f145a582cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37421 } } }
I20260925 02:26:33.823369 32607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:33.823534 32618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:33.824258 32618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:33.825616 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:33.814408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51441 (local address 127.0.0.1:37421)
0925 02:26:33.814618 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:33.814628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:33.814668 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:26:33.815198 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:33.815207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:33.815271 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:33.815433 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:33.816018 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:33.817400 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:33.818348 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:33.818973 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:33.819088 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:33.821674 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:33.821678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:33.821679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:33.821692 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:33.823851 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:33.824255 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:33.824258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:33.824258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:33.824415 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:33.824647 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:33.824648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:33.824648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:33.824982 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:33.825126 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:33.825284 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:26:33.825424 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260925 02:26:33.826224 32618 catalog_manager.cc:1347] Generated new cluster ID: aefec72766a7431f9515288a181e0bef
I20260925 02:26:33.826296 32618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:33.826583 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:33.834729 32618 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:33.835223 32618 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:33.835425 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:33.828098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43141 (local address 127.0.0.1:37421)
0925 02:26:33.828138 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:33.828140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:33.828218 (+ 78us) server_negotiation.cc:407] Connection header received
0925 02:26:33.828275 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:33.828276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:33.828286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:33.828338 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:33.829008 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:33.829704 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:33.830781 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:33.831091 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:33.831149 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:33.832483 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:33.832484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:33.832484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:33.832491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:33.834077 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:33.834476 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:33.834477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:33.834478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:33.834544 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:33.834819 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:33.834820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:33.834820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:33.834962 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:33.835033 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:33.835255 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:26:33.835340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:33.839643 32618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a06e7d4074a4c7aa41c0f145a582cff: Generated new TSK 0
I20260925 02:26:33.839820 32618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:33.844864 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32532
2026-09-25T02:26:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32632](info): set up 2 sockets
Sep 25 02:26:33 dist-test-slave-hj7s krb5kdc[32632](info): commencing operation
krb5kdc: starting...
W20260925 02:26:34.891211 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:34 dist-test-slave-hj7s krb5kdc[32632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:35.011593 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:35.093422 32645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:35.093639 32645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:35.093663 32645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:35.094784 32645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:35.094822 32645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:35.094836 32645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:35.094849 32645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:35.094861 32645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:35.096365 32645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:35.096560 32645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:35.096726 32645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:35.101585 32650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:35.101574 32651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:35.101574 32653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:35.101696 32645 server_base.cc:1034] running on GCE node
I20260925 02:26:35.102129 32645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:35.102501 32645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:35.103670 32645 hybrid_clock.cc:648] HybridClock initialized: now 1790303195103646 us; error 37 us; skew 500 ppm
Sep 25 02:26:35 dist-test-slave-hj7s krb5kdc[32632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:35.111074 32645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:35.111581 32645 webserver.cc:460] Webserver started at http://127.0.0.1:42069/ using document root <none> and password file <none>
I20260925 02:26:35.111810 32645 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:35.111884 32645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:35.112037 32645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:35.113193 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "897a5dc1e8c2417daaec740859920096"
format_stamp: "Formatted at 2026-09-25 02:26:35 on dist-test-slave-hj7s"
server_key: "6570e2199fb8c725a6a7f665b8287929"
server_key_iv: "e7f2b0a268eb9198a80323f7edfe93a9"
server_key_version: "encryptionkey@0"
I20260925 02:26:35.113564 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "897a5dc1e8c2417daaec740859920096"
format_stamp: "Formatted at 2026-09-25 02:26:35 on dist-test-slave-hj7s"
server_key: "6570e2199fb8c725a6a7f665b8287929"
server_key_iv: "e7f2b0a268eb9198a80323f7edfe93a9"
server_key_version: "encryptionkey@0"
I20260925 02:26:35.114956 32645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:35.115700 32660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:35.115888 32645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:35.116015 32645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "897a5dc1e8c2417daaec740859920096"
format_stamp: "Formatted at 2026-09-25 02:26:35 on dist-test-slave-hj7s"
server_key: "6570e2199fb8c725a6a7f665b8287929"
server_key_iv: "e7f2b0a268eb9198a80323f7edfe93a9"
server_key_version: "encryptionkey@0"
I20260925 02:26:35.116091 32645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:35.146401 32645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:35.148842 32645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:35.149051 32645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:35.153306 32645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46017
I20260925 02:26:35.153347 32713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46017 every 8 connection(s)
I20260925 02:26:35.153736 32645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:35.154640 32714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:35.156822 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32645
I20260925 02:26:35.156994 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:35.157059 32714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096: Bootstrap starting.
I20260925 02:26:35.157204 11764 external_mini_cluster.cc:1428] Setting key 4f5ac833b592ed0f8c8ddc4f92025303
I20260925 02:26:35.157820 32714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:35.158161 32714 log.cc:824] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:35.158993 32714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096: No bootstrap required, opened a new log
I20260925 02:26:35.161273 32714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 02:26:35.161418 32714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:35.161440 32714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897a5dc1e8c2417daaec740859920096, State: Initialized, Role: FOLLOWER
I20260925 02:26:35.161588 32714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 02:26:35.161648 32714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:35.161669 32714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:35.161736 32714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:35.162283 32714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 02:26:35.162379 32714 leader_election.cc:302] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897a5dc1e8c2417daaec740859920096; no voters:
I20260925 02:26:35.162585 32714 leader_election.cc:288] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:35.162607 32719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:35.162784 32719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [term 1 LEADER]: Becoming Leader. State: Replica: 897a5dc1e8c2417daaec740859920096, State: Running, Role: LEADER
I20260925 02:26:35.162931 32714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:35.162967 32719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } }
I20260925 02:26:35.163514 32720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897a5dc1e8c2417daaec740859920096. Latest consensus state: current_term: 1 leader_uuid: "897a5dc1e8c2417daaec740859920096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } } }
I20260925 02:26:35.163604 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:35.157730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46787 (local address 127.0.0.1:46017)
0925 02:26:35.157933 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:35.157945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:35.157983 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:26:35.158588 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:35.158598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:35.158664 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:35.158794 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:35.159036 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:35.159951 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:35.161175 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:35.161752 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:35.161891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:35.162693 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:35.162698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:35.162698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:35.162927 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:35.162971 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:35.162984 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:35.163214 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:35.163245 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 02:26:35.163313 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260925 02:26:35.163619 32720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:35.164201 32721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897a5dc1e8c2417daaec740859920096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897a5dc1e8c2417daaec740859920096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46017 } } }
I20260925 02:26:35.164286 32721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:35.164465 32728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:35.164583 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:35.165045 32728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:35.166827 32728 catalog_manager.cc:1347] Generated new cluster ID: e567ae995d8a41cd87031079f62070fa
I20260925 02:26:35.166880 32728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:35.169623 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:35.165976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46659 (local address 127.0.0.1:46017)
0925 02:26:35.166011 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:35.166013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:35.166307 (+ 294us) server_negotiation.cc:407] Connection header received
0925 02:26:35.166363 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:35.166364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:35.166376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:35.166423 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:35.166571 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:35.167272 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:35.168299 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:35.168633 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:35.168710 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:35.169385 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:35.169386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:35.169386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:35.169391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:35.169406 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:35.169410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:35.169473 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:35.169526 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:26:35.169549 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:35.177621 32728 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:35.178131 32728 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:35.180580 32728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897a5dc1e8c2417daaec740859920096: Generated new TSK 0
I20260925 02:26:35.180775 32728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:35.184110 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32645
2026-09-25T02:26:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:35 dist-test-slave-hj7s krb5kdc[32745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:35 dist-test-slave-hj7s krb5kdc[32745](info): set up 2 sockets
Sep 25 02:26:35 dist-test-slave-hj7s krb5kdc[32745](info): commencing operation
krb5kdc: starting...
W20260925 02:26:36.223053 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:36 dist-test-slave-hj7s krb5kdc[32745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:36.343762 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:36.429426 32758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:36.429653 32758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:36.429679 32758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:36.430984 32758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:36.431035 32758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:36.431051 32758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:36.431097 32758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:36.431129 32758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:36.432787 32758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:36.433090 32758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:36.433341 32758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:36.438766 32766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:36.438948 32758 server_base.cc:1034] running on GCE node
W20260925 02:26:36.438853 32763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:36.438771 32764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:36.439393 32758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:36.439745 32758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:36.440994 32758 hybrid_clock.cc:648] HybridClock initialized: now 1790303196440949 us; error 44 us; skew 500 ppm
Sep 25 02:26:36 dist-test-slave-hj7s krb5kdc[32745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:36.443683 32758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:36.444196 32758 webserver.cc:460] Webserver started at http://127.0.0.1:33341/ using document root <none> and password file <none>
I20260925 02:26:36.444408 32758 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:36.444453 32758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:36.444552 32758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:36.445755 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "73685938fd98439ba1fd16a3469b7e7d"
format_stamp: "Formatted at 2026-09-25 02:26:36 on dist-test-slave-hj7s"
server_key: "1850171411cc57f67770d3bd1772cda8"
server_key_iv: "ed7f6cf1de42fe53bff1ad06df36d370"
server_key_version: "encryptionkey@0"
I20260925 02:26:36.446090 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "73685938fd98439ba1fd16a3469b7e7d"
format_stamp: "Formatted at 2026-09-25 02:26:36 on dist-test-slave-hj7s"
server_key: "1850171411cc57f67770d3bd1772cda8"
server_key_iv: "ed7f6cf1de42fe53bff1ad06df36d370"
server_key_version: "encryptionkey@0"
I20260925 02:26:36.447474 32758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:36.448208 305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:36.448449 32758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:36.448550 32758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "73685938fd98439ba1fd16a3469b7e7d"
format_stamp: "Formatted at 2026-09-25 02:26:36 on dist-test-slave-hj7s"
server_key: "1850171411cc57f67770d3bd1772cda8"
server_key_iv: "ed7f6cf1de42fe53bff1ad06df36d370"
server_key_version: "encryptionkey@0"
I20260925 02:26:36.448609 32758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:36.451954 32758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:36.454299 32758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:36.454416 32758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:36.458370 32758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46599
I20260925 02:26:36.458494 357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46599 every 8 connection(s)
I20260925 02:26:36.458781 32758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:36.459695 358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:36.462193 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d: Bootstrap starting.
I20260925 02:26:36.462806 358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:36.463141 358 log.cc:824] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:36.463860 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d: No bootstrap required, opened a new log
I20260925 02:26:36.465654 358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260925 02:26:36.465830 358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:36.465871 358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73685938fd98439ba1fd16a3469b7e7d, State: Initialized, Role: FOLLOWER
I20260925 02:26:36.466048 358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260925 02:26:36.466135 358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:36.466180 358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:36.466235 358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:36.466864 358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260925 02:26:36.466991 358 leader_election.cc:302] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73685938fd98439ba1fd16a3469b7e7d; no voters:
I20260925 02:26:36.467199 358 leader_election.cc:288] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:36.467233 361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:36.467408 361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [term 1 LEADER]: Becoming Leader. State: Replica: 73685938fd98439ba1fd16a3469b7e7d, State: Running, Role: LEADER
I20260925 02:26:36.467509 358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:36.467548 361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } }
I20260925 02:26:36.467953 363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73685938fd98439ba1fd16a3469b7e7d. Latest consensus state: current_term: 1 leader_uuid: "73685938fd98439ba1fd16a3469b7e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } } }
I20260925 02:26:36.468091 362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73685938fd98439ba1fd16a3469b7e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73685938fd98439ba1fd16a3469b7e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46599 } } }
I20260925 02:26:36.468106 363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:36.468199 362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:36.468587 370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:36.468784 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 32758
I20260925 02:26:36.468950 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:36.469172 11764 external_mini_cluster.cc:1428] Setting key 327a3d3e3be67ddc5d5af9973d58e782
I20260925 02:26:36.469249 370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:36.470988 370 catalog_manager.cc:1347] Generated new cluster ID: ca32c5cae69444f885cfd0bd15c42a2f
I20260925 02:26:36.471041 370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:36.471660 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:36.469852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38445 (local address 127.0.0.1:46599)
0925 02:26:36.470011 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:36.470020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:36.470054 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:26:36.470175 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:36.470183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:36.470215 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:36.470659 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:36.471176 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:36.471179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:36.471179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:36.471183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:36.471217 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:36.471230 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:36.471294 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:36.471428 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:26:36.471472 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260925 02:26:36.472653 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:36.475173 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:36.474234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59895 (local address 127.0.0.1:46599)
0925 02:26:36.474270 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:36.474271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:36.474280 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:36.474300 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:36.474300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:36.474305 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:36.474371 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:36.474960 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:36.474961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:36.474961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:36.474965 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:36.474981 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:36.474985 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:36.475046 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:36.475098 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:26:36.475110 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:36.491698 370 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:36.491931 370 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:36.503587 370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73685938fd98439ba1fd16a3469b7e7d: Generated new TSK 0
I20260925 02:26:36.503798 370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:36.507655 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 32758
2026-09-25T02:26:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:36 dist-test-slave-hj7s krb5kdc[389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:36 dist-test-slave-hj7s krb5kdc[389](info): set up 2 sockets
Sep 25 02:26:36 dist-test-slave-hj7s krb5kdc[389](info): commencing operation
krb5kdc: starting...
W20260925 02:26:37.547225 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:37.668493 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:37.751771 402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:37.751987 402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:37.752010 402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:37.753175 402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:37.753216 402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:37.753232 402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:37.753244 402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:37.753257 402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:37.754675 402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:37.754878 402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:37.755040 402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:37.759868 408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:37.759887 410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:37.759878 407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:37.759938 402 server_base.cc:1034] running on GCE node
I20260925 02:26:37.760306 402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:37.760593 402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:37.761758 402 hybrid_clock.cc:648] HybridClock initialized: now 1790303197761754 us; error 37 us; skew 500 ppm
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:37.764228 402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:37.764694 402 webserver.cc:460] Webserver started at http://127.0.0.1:42639/ using document root <none> and password file <none>
I20260925 02:26:37.764968 402 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:37.765038 402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:37.765193 402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:37.766229 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "df33093ebf2a4fb6acca989f1846855b"
format_stamp: "Formatted at 2026-09-25 02:26:37 on dist-test-slave-hj7s"
server_key: "ca53173cfbfd19e2377440e43a612292"
server_key_iv: "507ada76a0275391533c7aafd5e8d25d"
server_key_version: "encryptionkey@0"
I20260925 02:26:37.766609 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "df33093ebf2a4fb6acca989f1846855b"
format_stamp: "Formatted at 2026-09-25 02:26:37 on dist-test-slave-hj7s"
server_key: "ca53173cfbfd19e2377440e43a612292"
server_key_iv: "507ada76a0275391533c7aafd5e8d25d"
server_key_version: "encryptionkey@0"
I20260925 02:26:37.767944 402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:37.768723 417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:37.768962 402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:26:37.769075 402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "df33093ebf2a4fb6acca989f1846855b"
format_stamp: "Formatted at 2026-09-25 02:26:37 on dist-test-slave-hj7s"
server_key: "ca53173cfbfd19e2377440e43a612292"
server_key_iv: "507ada76a0275391533c7aafd5e8d25d"
server_key_version: "encryptionkey@0"
I20260925 02:26:37.769167 402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:37.775354 402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:37.777127 402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:37.777282 402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:37.781116 402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33473
I20260925 02:26:37.781167 469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33473 every 8 connection(s)
I20260925 02:26:37.781527 402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:37.782363 470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:37.782972 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 402
I20260925 02:26:37.783104 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:37.783316 11764 external_mini_cluster.cc:1428] Setting key e0793d16d1d733c81d5e6ace104b08b8
I20260925 02:26:37.785218 470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b: Bootstrap starting.
I20260925 02:26:37.785775 470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:37.786070 470 log.cc:824] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:37.786774 470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b: No bootstrap required, opened a new log
I20260925 02:26:37.788813 470 raft_consensus.cc:348] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260925 02:26:37.789031 470 raft_consensus.cc:374] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:37.789103 470 raft_consensus.cc:729] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df33093ebf2a4fb6acca989f1846855b, State: Initialized, Role: FOLLOWER
I20260925 02:26:37.789318 470 consensus_queue.cc:260] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260925 02:26:37.789420 470 raft_consensus.cc:388] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:37.789466 470 raft_consensus.cc:482] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:37.789520 470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:26:37.790297 470 raft_consensus.cc:504] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260925 02:26:37.790412 470 leader_election.cc:302] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df33093ebf2a4fb6acca989f1846855b; no voters:
I20260925 02:26:37.790611 470 leader_election.cc:288] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:37.790652 475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:37.790838 475 raft_consensus.cc:686] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [term 1 LEADER]: Becoming Leader. State: Replica: df33093ebf2a4fb6acca989f1846855b, State: Running, Role: LEADER
I20260925 02:26:37.790912 470 sys_catalog.cc:564] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:37.790973 475 consensus_queue.cc:237] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } }
I20260925 02:26:37.791415 476 sys_catalog.cc:455] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df33093ebf2a4fb6acca989f1846855b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } } }
I20260925 02:26:37.791404 477 sys_catalog.cc:455] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df33093ebf2a4fb6acca989f1846855b. Latest consensus state: current_term: 1 leader_uuid: "df33093ebf2a4fb6acca989f1846855b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df33093ebf2a4fb6acca989f1846855b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33473 } } }
I20260925 02:26:37.791548 477 sys_catalog.cc:458] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:37.791535 476 sys_catalog.cc:458] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:37.793234 484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:37.793973 484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:37.794889 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:37.783824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:43863 (local address 127.0.0.1:33473)
0925 02:26:37.784029 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:37.784039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:37.784071 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:26:37.784513 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:37.784520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:37.784573 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:37.784717 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:37.785438 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:37.786567 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:37.787390 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:37.787829 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:37.787942 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:37.790979 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:37.790983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:37.790984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:37.790996 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:37.793166 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:37.793649 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:37.793651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:37.793651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:37.793783 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:37.794055 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:37.794056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:37.794056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:37.794296 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:37.794483 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:37.794566 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 02:26:37.794671 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260925 02:26:37.795573 484 catalog_manager.cc:1347] Generated new cluster ID: 49e114fa19b742789c25bf572c7fe57e
I20260925 02:26:37.795627 484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:37.795943 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:37.803540 484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:37.804245 484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:37.807812 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:37.801309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:35923 (local address 127.0.0.1:33473)
0925 02:26:37.801347 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:37.801349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:37.801468 (+ 119us) server_negotiation.cc:407] Connection header received
0925 02:26:37.801541 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:37.801542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:37.801552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:37.801600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:37.802152 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:37.802657 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:37.803587 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:37.803832 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:37.803909 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:37.805171 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:37.805172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:37.805173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:37.805181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:37.806577 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:37.806949 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:37.806950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:37.806950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:37.807007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:37.807235 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:37.807236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:37.807237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:37.807346 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:37.807518 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:37.807612 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 02:26:37.807693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:26:37.813271 484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df33093ebf2a4fb6acca989f1846855b: Generated new TSK 0
I20260925 02:26:37.813437 484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:37.815187 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 402
W20260925 02:26:37.820156 387 connection.cc:535] client connection to 127.0.0.1:33473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33473): Connection reset by peer (error 104)
2026-09-25T02:26:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[502](info): set up 2 sockets
Sep 25 02:26:37 dist-test-slave-hj7s krb5kdc[502](info): commencing operation
krb5kdc: starting...
W20260925 02:26:38.859355 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:38 dist-test-slave-hj7s krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:38.976361 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:39.058351 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:39.058540 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:39.058562 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:39.059617 515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:39.059651 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:39.059665 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:39.059677 515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:39.059689 515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:39.061137 515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:39.061324 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:39.061491 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:39.066350 521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:39.066322 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:39.066431 515 server_base.cc:1034] running on GCE node
W20260925 02:26:39.066326 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:39.066777 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:39.067006 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:39.068145 515 hybrid_clock.cc:648] HybridClock initialized: now 1790303199068124 us; error 35 us; skew 500 ppm
Sep 25 02:26:39 dist-test-slave-hj7s krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:39.070572 515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:39.071054 515 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
I20260925 02:26:39.071290 515 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:39.071332 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:39.071522 515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:39.072530 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "1f690aced8aa4494bf2d734e020cd158"
format_stamp: "Formatted at 2026-09-25 02:26:39 on dist-test-slave-hj7s"
server_key: "9ef055af69d276e2cc132a0c4e44da02"
server_key_iv: "7929d65664b95c963a2dc42f2b877c18"
server_key_version: "encryptionkey@0"
I20260925 02:26:39.072942 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "1f690aced8aa4494bf2d734e020cd158"
format_stamp: "Formatted at 2026-09-25 02:26:39 on dist-test-slave-hj7s"
server_key: "9ef055af69d276e2cc132a0c4e44da02"
server_key_iv: "7929d65664b95c963a2dc42f2b877c18"
server_key_version: "encryptionkey@0"
I20260925 02:26:39.074332 515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:39.075063 531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:39.075249 515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:39.075358 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "1f690aced8aa4494bf2d734e020cd158"
format_stamp: "Formatted at 2026-09-25 02:26:39 on dist-test-slave-hj7s"
server_key: "9ef055af69d276e2cc132a0c4e44da02"
server_key_iv: "7929d65664b95c963a2dc42f2b877c18"
server_key_version: "encryptionkey@0"
I20260925 02:26:39.075433 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:39.103068 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:39.105036 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:39.105193 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:39.109349 515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46149
I20260925 02:26:39.109380 583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46149 every 8 connection(s)
I20260925 02:26:39.109740 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:39.110615 584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:39.111413 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 515
I20260925 02:26:39.111572 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:39.111748 11764 external_mini_cluster.cc:1428] Setting key b4da7f8543f85cc8e6390026646ef028
I20260925 02:26:39.113626 584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158: Bootstrap starting.
I20260925 02:26:39.114229 584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:39.114600 584 log.cc:824] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:39.115365 584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158: No bootstrap required, opened a new log
I20260925 02:26:39.117375 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:39.112209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:58721 (local address 127.0.0.1:46149)
0925 02:26:39.112415 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:39.112426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:39.112465 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:26:39.112881 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:39.112887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:39.113022 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:39.113159 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:39.113403 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:39.114303 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:39.115464 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:39.115870 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:39.115951 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:39.116634 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:39.116638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:39.116639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:39.116810 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:39.116839 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:39.116852 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:39.116975 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:39.117093 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:26:39.117134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260925 02:26:39.117715 584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260925 02:26:39.117847 584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:39.117869 584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f690aced8aa4494bf2d734e020cd158, State: Initialized, Role: FOLLOWER
I20260925 02:26:39.118005 584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260925 02:26:39.118104 584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:39.118141 584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:39.118170 584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:39.118458 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:39.118806 584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260925 02:26:39.118918 584 leader_election.cc:302] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f690aced8aa4494bf2d734e020cd158; no voters:
I20260925 02:26:39.119083 584 leader_election.cc:288] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:39.119138 589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:39.119323 589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [term 1 LEADER]: Becoming Leader. State: Replica: 1f690aced8aa4494bf2d734e020cd158, State: Running, Role: LEADER
I20260925 02:26:39.119342 584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:39.119442 589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } }
I20260925 02:26:39.119792 589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f690aced8aa4494bf2d734e020cd158. Latest consensus state: current_term: 1 leader_uuid: "1f690aced8aa4494bf2d734e020cd158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260925 02:26:39.119805 590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f690aced8aa4494bf2d734e020cd158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f690aced8aa4494bf2d734e020cd158" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46149 } } }
I20260925 02:26:39.119941 589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:39.119952 590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:39.120244 604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:39.120956 604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:39.123608 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:39.120156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:44043 (local address 127.0.0.1:46149)
0925 02:26:39.120188 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:39.120190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:39.120680 (+ 490us) server_negotiation.cc:407] Connection header received
0925 02:26:39.120749 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:39.120750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:39.120761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:39.120812 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:39.121127 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:39.121648 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:39.122465 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:39.122681 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:39.122765 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:39.123388 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:39.123389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:39.123390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:39.123394 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:39.123408 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:39.123412 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:39.123464 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:39.123525 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:26:39.123534 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:26:39.126060 604 catalog_manager.cc:1347] Generated new cluster ID: ce702b692cec43c48aa3eb7d9a8a9a88
I20260925 02:26:39.126113 604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:39.142414 604 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:39.143002 604 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:39.146991 604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f690aced8aa4494bf2d734e020cd158: Generated new TSK 0
I20260925 02:26:39.147193 604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:39.152563 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 515
W20260925 02:26:39.157119 500 connection.cc:535] client connection to 127.0.0.1:46149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46149): Connection reset by peer (error 104)
2026-09-25T02:26:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:39 dist-test-slave-hj7s krb5kdc[616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:39 dist-test-slave-hj7s krb5kdc[616](info): set up 2 sockets
Sep 25 02:26:39 dist-test-slave-hj7s krb5kdc[616](info): commencing operation
krb5kdc: starting...
W20260925 02:26:40.191208 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:40 dist-test-slave-hj7s krb5kdc[616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:40.312101 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:40.394933 629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:40.395157 629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:40.395181 629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:40.396394 629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:40.396440 629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:40.396456 629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:40.396513 629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:40.396533 629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:40.398113 629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:40.398381 629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:40.398627 629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:40.403602 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:40.403610 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:40.403656 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:40.403779 629 server_base.cc:1034] running on GCE node
I20260925 02:26:40.404170 629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:40.404458 629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:40.405653 629 hybrid_clock.cc:648] HybridClock initialized: now 1790303200405598 us; error 30 us; skew 500 ppm
Sep 25 02:26:40 dist-test-slave-hj7s krb5kdc[616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:40.414693 629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:40.415216 629 webserver.cc:460] Webserver started at http://127.0.0.1:42533/ using document root <none> and password file <none>
I20260925 02:26:40.415444 629 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:40.415491 629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:40.415638 629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:40.416754 629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897"
format_stamp: "Formatted at 2026-09-25 02:26:40 on dist-test-slave-hj7s"
server_key: "4f80d3d64598d320e0fccd3c788198fc"
server_key_iv: "539b71446cc581ed19e99af9347b6b4d"
server_key_version: "encryptionkey@0"
I20260925 02:26:40.417191 629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897"
format_stamp: "Formatted at 2026-09-25 02:26:40 on dist-test-slave-hj7s"
server_key: "4f80d3d64598d320e0fccd3c788198fc"
server_key_iv: "539b71446cc581ed19e99af9347b6b4d"
server_key_version: "encryptionkey@0"
I20260925 02:26:40.418582 629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:40.419622 644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:40.419813 629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:40.419934 629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897"
format_stamp: "Formatted at 2026-09-25 02:26:40 on dist-test-slave-hj7s"
server_key: "4f80d3d64598d320e0fccd3c788198fc"
server_key_iv: "539b71446cc581ed19e99af9347b6b4d"
server_key_version: "encryptionkey@0"
I20260925 02:26:40.420010 629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:40.423458 629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:40.425408 629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:40.425582 629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:40.429651 629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39597
I20260925 02:26:40.429704 697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39597 every 8 connection(s)
I20260925 02:26:40.430063 629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:40.430944 698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:40.433318 698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897: Bootstrap starting.
I20260925 02:26:40.433869 698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:40.434122 698 log.cc:824] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:40.434751 698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897: No bootstrap required, opened a new log
I20260925 02:26:40.436321 698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260925 02:26:40.436461 698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:40.436486 698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00eae26b0d4f46f8bfd48fe7cf2c0897, State: Initialized, Role: FOLLOWER
I20260925 02:26:40.436589 698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260925 02:26:40.436640 698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:40.436661 698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:40.436688 698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:40.437193 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 629
I20260925 02:26:40.437345 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:40.437284 698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260925 02:26:40.437446 698 leader_election.cc:302] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00eae26b0d4f46f8bfd48fe7cf2c0897; no voters:
I20260925 02:26:40.437565 11764 external_mini_cluster.cc:1428] Setting key 65aaf9fc6fb2f90acad6e71652abb2d6
I20260925 02:26:40.437649 698 leader_election.cc:288] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:40.437690 701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:40.437873 701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [term 1 LEADER]: Becoming Leader. State: Replica: 00eae26b0d4f46f8bfd48fe7cf2c0897, State: Running, Role: LEADER
I20260925 02:26:40.437943 698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:40.438050 701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260925 02:26:40.438478 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } } }
I20260925 02:26:40.438583 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:40.438767 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00eae26b0d4f46f8bfd48fe7cf2c0897. Latest consensus state: current_term: 1 leader_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00eae26b0d4f46f8bfd48fe7cf2c0897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } } }
I20260925 02:26:40.438995 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:40.438930 711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:40.439677 711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:40.440768 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:40.438688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:58779 (local address 127.0.0.1:39597)
0925 02:26:40.438863 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:40.438874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:40.438907 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:26:40.439049 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:40.439064 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:40.439097 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:40.439532 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:40.440104 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:40.440107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:40.440108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:40.440112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:40.440176 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:40.440202 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:40.440274 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:40.440467 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 02:26:40.440569 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260925 02:26:40.441533 711 catalog_manager.cc:1347] Generated new cluster ID: d061d1db3c77467b89116409032b7bc6
I20260925 02:26:40.441588 711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:40.441985 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:40.444629 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:40.443462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:38917 (local address 127.0.0.1:39597)
0925 02:26:40.443510 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:40.443512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:40.443630 (+ 118us) server_negotiation.cc:407] Connection header received
0925 02:26:40.443679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:40.443680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:40.443685 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:40.443766 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:40.444395 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:40.444396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:40.444396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:40.444401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:40.444420 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:40.444426 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:40.444496 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:40.444539 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 02:26:40.444553 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 02:26:40.452175 711 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:40.452363 711 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:40.459209 711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00eae26b0d4f46f8bfd48fe7cf2c0897: Generated new TSK 0
I20260925 02:26:40.459421 711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:40.465565 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 629
2026-09-25T02:26:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:40 dist-test-slave-hj7s krb5kdc[729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:40 dist-test-slave-hj7s krb5kdc[729](info): set up 2 sockets
Sep 25 02:26:40 dist-test-slave-hj7s krb5kdc[729](info): commencing operation
krb5kdc: starting...
W20260925 02:26:41.515429 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:41.639849 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:41.722751 742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:41.722973 742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:41.723012 742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:41.724161 742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:41.724224 742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:41.724249 742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:41.724270 742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:41.724298 742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:41.725781 742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:41.726047 742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:41.726279 742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:41.731122 748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:41.731105 750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:41.731105 747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:41.731292 742 server_base.cc:1034] running on GCE node
I20260925 02:26:41.731528 742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:41.731765 742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:41.732964 742 hybrid_clock.cc:648] HybridClock initialized: now 1790303201732945 us; error 27 us; skew 500 ppm
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:41.735666 742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:41.736191 742 webserver.cc:460] Webserver started at http://127.0.0.1:36077/ using document root <none> and password file <none>
I20260925 02:26:41.736454 742 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:41.736522 742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:41.736675 742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:41.738066 742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "32472e46be88444c9a61435600c1e22f"
format_stamp: "Formatted at 2026-09-25 02:26:41 on dist-test-slave-hj7s"
server_key: "99b9e95ba08419f9036cbd6eb6defe0f"
server_key_iv: "a1f7c525588a984238aaa3099a6c0d83"
server_key_version: "encryptionkey@0"
I20260925 02:26:41.738464 742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "32472e46be88444c9a61435600c1e22f"
format_stamp: "Formatted at 2026-09-25 02:26:41 on dist-test-slave-hj7s"
server_key: "99b9e95ba08419f9036cbd6eb6defe0f"
server_key_iv: "a1f7c525588a984238aaa3099a6c0d83"
server_key_version: "encryptionkey@0"
I20260925 02:26:41.739769 742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:41.740434 757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:41.740622 742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:41.740697 742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "32472e46be88444c9a61435600c1e22f"
format_stamp: "Formatted at 2026-09-25 02:26:41 on dist-test-slave-hj7s"
server_key: "99b9e95ba08419f9036cbd6eb6defe0f"
server_key_iv: "a1f7c525588a984238aaa3099a6c0d83"
server_key_version: "encryptionkey@0"
I20260925 02:26:41.740748 742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:41.760983 742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:41.762709 742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:41.762820 742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:41.766772 742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40275
I20260925 02:26:41.766829 809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40275 every 8 connection(s)
I20260925 02:26:41.767172 742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:41.768028 810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:41.770377 810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f: Bootstrap starting.
I20260925 02:26:41.771016 810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:41.771312 810 log.cc:824] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:41.771996 810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f: No bootstrap required, opened a new log
I20260925 02:26:41.773602 810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } }
I20260925 02:26:41.773741 810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:41.773783 810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32472e46be88444c9a61435600c1e22f, State: Initialized, Role: FOLLOWER
I20260925 02:26:41.773964 810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } }
I20260925 02:26:41.774045 810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:41.774091 810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:41.774145 810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:41.774675 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 742
I20260925 02:26:41.774770 810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } }
I20260925 02:26:41.774816 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:41.774889 810 leader_election.cc:302] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32472e46be88444c9a61435600c1e22f; no voters:
I20260925 02:26:41.775020 11764 external_mini_cluster.cc:1428] Setting key b393c3718aae33d3294697449cf4d425
I20260925 02:26:41.775063 810 leader_election.cc:288] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:41.775112 814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:41.775261 814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [term 1 LEADER]: Becoming Leader. State: Replica: 32472e46be88444c9a61435600c1e22f, State: Running, Role: LEADER
I20260925 02:26:41.775333 810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:41.775355 814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } }
I20260925 02:26:41.775660 814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32472e46be88444c9a61435600c1e22f. Latest consensus state: current_term: 1 leader_uuid: "32472e46be88444c9a61435600c1e22f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } } }
I20260925 02:26:41.775743 814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:41.775986 816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32472e46be88444c9a61435600c1e22f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32472e46be88444c9a61435600c1e22f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40275 } } }
I20260925 02:26:41.776090 816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:41.776825 821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:41.777539 821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:41.781314 821 catalog_manager.cc:1347] Generated new cluster ID: b3822b8e5a5e4e2fb7aba8e9739bb977
I20260925 02:26:41.781374 821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:26:41.787161 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:41.775677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:35311 (local address 127.0.0.1:40275)
0925 02:26:41.775870 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:41.775880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:41.776196 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:26:41.776318 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:41.776325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:41.776402 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:41.776559 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:41.777626 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:41.778860 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:41.779956 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:41.780373 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:41.780453 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:41.783291 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:41.783295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:41.783296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:41.783308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:41.785436 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:41.785879 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:41.785883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:41.785883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:41.786012 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:41.786240 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:41.786241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:41.786241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:41.786571 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:41.786748 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:41.786836 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:26:41.786944 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
I20260925 02:26:41.788170 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:41.790153 821 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:41.790827 821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:41.795962 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:41.789655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:47251 (local address 127.0.0.1:40275)
0925 02:26:41.789705 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:41.789708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:41.790007 (+ 299us) server_negotiation.cc:407] Connection header received
0925 02:26:41.790102 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:41.790103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:41.790114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:41.790175 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:41.790747 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:41.791269 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:41.791970 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:41.792186 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:41.792266 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:41.793487 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:41.793489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:41.793489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:41.793495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:41.794812 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:41.795171 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:41.795172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:41.795173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:41.795229 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:41.795448 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:41.795449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:41.795450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:41.795554 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:41.795657 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:41.795777 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:26:41.795851 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:41.800276 821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32472e46be88444c9a61435600c1e22f: Generated new TSK 0
I20260925 02:26:41.800467 821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:41.802870 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 742
W20260925 02:26:41.807056 727 connection.cc:535] client connection to 127.0.0.1:40275 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40275): Connection reset by peer (error 104)
2026-09-25T02:26:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[842](info): set up 2 sockets
Sep 25 02:26:41 dist-test-slave-hj7s krb5kdc[842](info): commencing operation
krb5kdc: starting...
W20260925 02:26:42.851421 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:42 dist-test-slave-hj7s krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:42.971513 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:43.054138 855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:43.054399 855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:43.054440 855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:43.055626 855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:43.055693 855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:43.055718 855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:43.055739 855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:43.055768 855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:43.057310 855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:43.057606 855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:43.057837 855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:43.062774 861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:43.062876 863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:43.062861 855 server_base.cc:1034] running on GCE node
W20260925 02:26:43.062789 860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:43.063305 855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:43.063558 855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:43.064721 855 hybrid_clock.cc:648] HybridClock initialized: now 1790303203064715 us; error 32 us; skew 500 ppm
Sep 25 02:26:43 dist-test-slave-hj7s krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:43.067137 855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:43.067610 855 webserver.cc:460] Webserver started at http://127.0.0.1:43741/ using document root <none> and password file <none>
I20260925 02:26:43.067833 855 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:43.067905 855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:43.068053 855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:43.069208 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "b00e6769e37a4ae7a4aacd41536af816"
format_stamp: "Formatted at 2026-09-25 02:26:43 on dist-test-slave-hj7s"
server_key: "75e6e35399ed952aebf7c71300fc92ce"
server_key_iv: "bc7f979b39d90f790b9c27cab932b6f7"
server_key_version: "encryptionkey@0"
I20260925 02:26:43.069579 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "b00e6769e37a4ae7a4aacd41536af816"
format_stamp: "Formatted at 2026-09-25 02:26:43 on dist-test-slave-hj7s"
server_key: "75e6e35399ed952aebf7c71300fc92ce"
server_key_iv: "bc7f979b39d90f790b9c27cab932b6f7"
server_key_version: "encryptionkey@0"
I20260925 02:26:43.070955 855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:43.071702 870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:43.071884 855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:43.071991 855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "b00e6769e37a4ae7a4aacd41536af816"
format_stamp: "Formatted at 2026-09-25 02:26:43 on dist-test-slave-hj7s"
server_key: "75e6e35399ed952aebf7c71300fc92ce"
server_key_iv: "bc7f979b39d90f790b9c27cab932b6f7"
server_key_version: "encryptionkey@0"
I20260925 02:26:43.072065 855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:43.087002 855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:43.088979 855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:43.089162 855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:43.093357 855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36783
I20260925 02:26:43.093397 922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36783 every 8 connection(s)
I20260925 02:26:43.093876 855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:43.094934 923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:43.096354 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 855
I20260925 02:26:43.096483 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:43.096705 11764 external_mini_cluster.cc:1428] Setting key 5fccc979b3c7bf00c1dded392ad6b8e4
I20260925 02:26:43.097780 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: Bootstrap starting.
I20260925 02:26:43.098492 923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:43.098793 923 log.cc:824] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:43.099481 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: No bootstrap required, opened a new log
I20260925 02:26:43.101912 923 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260925 02:26:43.102049 923 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:43.102072 923 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00e6769e37a4ae7a4aacd41536af816, State: Initialized, Role: FOLLOWER
I20260925 02:26:43.102223 923 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260925 02:26:43.102304 923 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:43.102385 923 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:43.102442 923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:43.102689 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:43.097305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:35963 (local address 127.0.0.1:36783)
0925 02:26:43.097498 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:43.097506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:43.097532 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:26:43.097990 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:43.097997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:43.098058 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:43.098179 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:43.098432 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:43.099451 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:43.100557 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:43.101039 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:43.101122 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:43.101862 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:43.101866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:43.101866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:43.102031 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:43.102093 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:43.102111 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:43.102217 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:43.102480 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:26:43.102515 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260925 02:26:43.103106 923 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260925 02:26:43.103215 923 leader_election.cc:302] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00e6769e37a4ae7a4aacd41536af816; no voters:
I20260925 02:26:43.103380 923 leader_election.cc:288] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:43.103403 928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:43.103515 928 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [term 1 LEADER]: Becoming Leader. State: Replica: b00e6769e37a4ae7a4aacd41536af816, State: Running, Role: LEADER
I20260925 02:26:43.103644 923 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:43.103647 928 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } }
I20260925 02:26:43.103739 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:43.104086 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00e6769e37a4ae7a4aacd41536af816. Latest consensus state: current_term: 1 leader_uuid: "b00e6769e37a4ae7a4aacd41536af816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } } }
I20260925 02:26:43.104169 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:43.104327 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00e6769e37a4ae7a4aacd41536af816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00e6769e37a4ae7a4aacd41536af816" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36783 } } }
I20260925 02:26:43.104389 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816 [sys.catalog]: This master's current role is: LEADER
W20260925 02:26:43.105558 946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:43.105666 946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:43.105767 935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:43.105927 935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:43.107335 935 catalog_manager.cc:1347] Generated new cluster ID: 103e680abbc945f8b35ee9f7fc528ed2
I20260925 02:26:43.107385 935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:43.109797 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:43.105589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:39561 (local address 127.0.0.1:36783)
0925 02:26:43.105624 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:43.105626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:43.106380 (+ 754us) server_negotiation.cc:407] Connection header received
0925 02:26:43.106501 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:43.106502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:43.106513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:43.106560 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:43.106693 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:43.107395 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:43.108402 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:43.108745 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:43.108818 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:43.109547 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:43.109549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:43.109549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:43.109554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:43.109571 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:43.109577 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:43.109647 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:43.109712 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:26:43.109724 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 02:26:43.124660 935 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:43.125492 935 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:43.136245 935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00e6769e37a4ae7a4aacd41536af816: Generated new TSK 0
I20260925 02:26:43.136504 935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:43.147534 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 855
W20260925 02:26:43.152347 840 connection.cc:535] client connection to 127.0.0.1:36783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36783): Connection reset by peer (error 104)
2026-09-25T02:26:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:43 dist-test-slave-hj7s krb5kdc[955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:43 dist-test-slave-hj7s krb5kdc[955](info): set up 2 sockets
Sep 25 02:26:43 dist-test-slave-hj7s krb5kdc[955](info): commencing operation
krb5kdc: starting...
W20260925 02:26:44.191150 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:44 dist-test-slave-hj7s krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:26:44.308485 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:44.390080 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:44.390278 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:44.390301 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:44.391391 968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:44.391426 968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:44.391441 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:44.391453 968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:44.391465 968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:44.392866 968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:44.393172 968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:44.393422 968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:44.398339 973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:44.398368 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:44.398437 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:44.398551 968 server_base.cc:1034] running on GCE node
I20260925 02:26:44.398824 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:44.399164 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:44.400322 968 hybrid_clock.cc:648] HybridClock initialized: now 1790303204400298 us; error 43 us; skew 500 ppm
Sep 25 02:26:44 dist-test-slave-hj7s krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:44.402833 968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:26:44.403316 968 webserver.cc:460] Webserver started at http://127.0.0.1:39019/ using document root <none> and password file <none>
I20260925 02:26:44.403564 968 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:44.403631 968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:44.403784 968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:44.404852 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "b3e2528972c14a62826f3acefeac3253"
format_stamp: "Formatted at 2026-09-25 02:26:44 on dist-test-slave-hj7s"
server_key: "8f5260ece2c7fb17393639edec939f5e"
server_key_iv: "c982d56a9895bc8f33385ec909651feb"
server_key_version: "encryptionkey@0"
I20260925 02:26:44.405287 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "b3e2528972c14a62826f3acefeac3253"
format_stamp: "Formatted at 2026-09-25 02:26:44 on dist-test-slave-hj7s"
server_key: "8f5260ece2c7fb17393639edec939f5e"
server_key_iv: "c982d56a9895bc8f33385ec909651feb"
server_key_version: "encryptionkey@0"
I20260925 02:26:44.406725 968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:44.407436 983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:44.407591 968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:44.407729 968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "b3e2528972c14a62826f3acefeac3253"
format_stamp: "Formatted at 2026-09-25 02:26:44 on dist-test-slave-hj7s"
server_key: "8f5260ece2c7fb17393639edec939f5e"
server_key_iv: "c982d56a9895bc8f33385ec909651feb"
server_key_version: "encryptionkey@0"
I20260925 02:26:44.407811 968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:44.411175 968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:44.413062 968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:44.413216 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:44.417333 968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40171
I20260925 02:26:44.417366 1035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40171 every 8 connection(s)
I20260925 02:26:44.417697 968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:44.418565 1036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:44.420738 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: Bootstrap starting.
I20260925 02:26:44.421351 1036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:44.421588 1036 log.cc:824] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:44.422195 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: No bootstrap required, opened a new log
I20260925 02:26:44.423018 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 968
I20260925 02:26:44.423151 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:44.423331 11764 external_mini_cluster.cc:1428] Setting key a5784ac6c8edd13d131c13c7c6b9b574
I20260925 02:26:44.424199 1036 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260925 02:26:44.424381 1036 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:44.424461 1036 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e2528972c14a62826f3acefeac3253, State: Initialized, Role: FOLLOWER
I20260925 02:26:44.424594 1036 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260925 02:26:44.425336 1036 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:44.425397 1036 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:44.425463 1036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:44.425805 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:44.423863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:46147 (local address 127.0.0.1:40171)
0925 02:26:44.424090 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:44.424100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:44.424136 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:26:44.424247 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:44.424254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:44.424284 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:44.424440 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:44.425192 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:44.425196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:44.425197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:44.425202 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:44.425261 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:44.425281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:44.425349 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:44.425561 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 02:26:44.425610 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":206,"threads_started":1}
I20260925 02:26:44.426160 1036 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260925 02:26:44.426263 1036 leader_election.cc:302] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e2528972c14a62826f3acefeac3253; no voters:
I20260925 02:26:44.426445 1036 leader_election.cc:288] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:44.426484 1041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:44.426630 1041 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [term 1 LEADER]: Becoming Leader. State: Replica: b3e2528972c14a62826f3acefeac3253, State: Running, Role: LEADER
I20260925 02:26:44.426707 1036 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:44.426827 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:44.426815 1041 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } }
I20260925 02:26:44.427178 1041 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e2528972c14a62826f3acefeac3253. Latest consensus state: current_term: 1 leader_uuid: "b3e2528972c14a62826f3acefeac3253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } } }
I20260925 02:26:44.427251 1041 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:44.427193 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e2528972c14a62826f3acefeac3253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e2528972c14a62826f3acefeac3253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40171 } } }
I20260925 02:26:44.427305 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253 [sys.catalog]: This master's current role is: LEADER
W20260925 02:26:44.428543 1061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:44.428609 1061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:44.428745 1049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:44.428973 1049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:44.429759 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:44.428492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:51095 (local address 127.0.0.1:40171)
0925 02:26:44.428527 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:44.428529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:44.428777 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:26:44.428874 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:44.428875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:44.428882 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:44.428975 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:44.429553 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:44.429555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:44.429555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:44.429559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:44.429574 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:44.429578 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:44.429625 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:44.429679 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:26:44.429694 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:44.430442 1049 catalog_manager.cc:1347] Generated new cluster ID: 6d536216c7df419c8c97e57bb1a462d8
I20260925 02:26:44.430505 1049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:44.453774 1049 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:44.453996 1049 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:44.463356 1049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e2528972c14a62826f3acefeac3253: Generated new TSK 0
I20260925 02:26:44.463609 1049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:44.467183 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 968
2026-09-25T02:26:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:44 dist-test-slave-hj7s krb5kdc[1067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:44 dist-test-slave-hj7s krb5kdc[1067](info): set up 2 sockets
Sep 25 02:26:44 dist-test-slave-hj7s krb5kdc[1067](info): commencing operation
krb5kdc: starting...
W20260925 02:26:45.507269 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:45.628800 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35537
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:45.710438 1080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:45.710783 1080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:45.710837 1080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:45.712186 1080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:45.712262 1080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:45.712293 1080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:45.712328 1080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:45.712361 1080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:45.714074 1080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:35537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:45.714418 1080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:45.714697 1080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:45.720120 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:45.720085 1088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:45.720290 1080 server_base.cc:1034] running on GCE node
W20260925 02:26:45.720077 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:45.720664 1080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:45.720957 1080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:45.722106 1080 hybrid_clock.cc:648] HybridClock initialized: now 1790303205722084 us; error 40 us; skew 500 ppm
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303205, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:45.739423 1080 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:45.739934 1080 webserver.cc:460] Webserver started at http://127.11.125.62:37035/ using document root <none> and password file <none>
I20260925 02:26:45.740178 1080 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:45.740248 1080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:45.740404 1080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:45.741605 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "8cbec1ab218d4d80b8e72289365bf63a"
format_stamp: "Formatted at 2026-09-25 02:26:45 on dist-test-slave-hj7s"
server_key: "c4fcb540fa34c3df6b7e0d66a474efca"
server_key_iv: "7e9deadc8a8213f162cfe6372574c6ad"
server_key_version: "encryptionkey@0"
I20260925 02:26:45.741999 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "8cbec1ab218d4d80b8e72289365bf63a"
format_stamp: "Formatted at 2026-09-25 02:26:45 on dist-test-slave-hj7s"
server_key: "c4fcb540fa34c3df6b7e0d66a474efca"
server_key_iv: "7e9deadc8a8213f162cfe6372574c6ad"
server_key_version: "encryptionkey@0"
I20260925 02:26:45.743428 1080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:26:45.744251 1096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:45.744470 1080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:45.744586 1080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "8cbec1ab218d4d80b8e72289365bf63a"
format_stamp: "Formatted at 2026-09-25 02:26:45 on dist-test-slave-hj7s"
server_key: "c4fcb540fa34c3df6b7e0d66a474efca"
server_key_iv: "7e9deadc8a8213f162cfe6372574c6ad"
server_key_version: "encryptionkey@0"
I20260925 02:26:45.744660 1080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:45.767642 1080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:45.769551 1080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:45.769711 1080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:45.773729 1080 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:35537
I20260925 02:26:45.773772 1148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:35537 every 8 connection(s)
I20260925 02:26:45.774138 1080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:45.775005 1149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:45.777370 1149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a: Bootstrap starting.
I20260925 02:26:45.777951 1149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:45.778239 1149 log.cc:824] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:45.778911 1149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a: No bootstrap required, opened a new log
I20260925 02:26:45.780490 1149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } }
I20260925 02:26:45.780645 1149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:45.780702 1149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbec1ab218d4d80b8e72289365bf63a, State: Initialized, Role: FOLLOWER
I20260925 02:26:45.780869 1149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } }
I20260925 02:26:45.780984 1149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:45.781037 1149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:45.781095 1149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:45.781899 1149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } }
I20260925 02:26:45.782016 1149 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbec1ab218d4d80b8e72289365bf63a; no voters:
I20260925 02:26:45.782176 1149 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:45.782198 1152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:45.782341 1152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [term 1 LEADER]: Becoming Leader. State: Replica: 8cbec1ab218d4d80b8e72289365bf63a, State: Running, Role: LEADER
I20260925 02:26:45.782455 1149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:45.782475 1152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } }
I20260925 02:26:45.782871 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbec1ab218d4d80b8e72289365bf63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } } }
I20260925 02:26:45.782881 1154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbec1ab218d4d80b8e72289365bf63a. Latest consensus state: current_term: 1 leader_uuid: "8cbec1ab218d4d80b8e72289365bf63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbec1ab218d4d80b8e72289365bf63a" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35537 } } }
I20260925 02:26:45.783030 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:45.783037 1154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:45.783478 1161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:45.783664 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1080
I20260925 02:26:45.783792 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:45.784020 11764 external_mini_cluster.cc:1428] Setting key eed69f6ad01ee9f54154274c8e5ec5e0
I20260925 02:26:45.784024 1161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:45.785775 1161 catalog_manager.cc:1347] Generated new cluster ID: 93e852c9a2e94468adb7b1ea60cc3a0f
I20260925 02:26:45.785827 1161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:45.795920 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:45.784454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52607 (local address 127.11.125.62:35537)
0925 02:26:45.784821 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:45.784830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:45.784862 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:26:45.784993 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:45.785000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:45.785066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:45.785216 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:45.786059 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:45.787461 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:45.788502 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:45.789218 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:45.789336 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:45.792196 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:45.792200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:45.792200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:45.792212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:45.794131 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:45.794623 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:45.794626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:45.794626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:45.794732 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:45.794978 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:45.794979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:45.794979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:45.795278 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:45.795421 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:45.795582 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:26:45.795686 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":108,"threads_started":1}
I20260925 02:26:45.796919 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:45.806457 1161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:45.807476 1161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:45.808699 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:45.801206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57809 (local address 127.11.125.62:35537)
0925 02:26:45.801247 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:45.801249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:45.801420 (+ 171us) server_negotiation.cc:407] Connection header received
0925 02:26:45.801486 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:45.801486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:45.801498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:45.801553 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:45.802165 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:45.802956 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:45.803910 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:45.804258 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:45.804330 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:45.805746 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:45.805748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:45.805749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:45.805756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:45.807434 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:45.807868 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:45.807869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:45.807869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:45.807927 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:45.808169 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:45.808170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:45.808170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:45.808285 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:45.808395 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:45.808546 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:26:45.808609 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:45.813673 1161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbec1ab218d4d80b8e72289365bf63a: Generated new TSK 0
I20260925 02:26:45.813868 1161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:45.818974 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1080
W20260925 02:26:45.823354 1065 connection.cc:535] client connection to 127.11.125.62:35537 recv error: Network error: failed to read from TLS socket (remote: 127.11.125.62:35537): Connection reset by peer (error 104)
2026-09-25T02:26:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1180](info): set up 2 sockets
Sep 25 02:26:45 dist-test-slave-hj7s krb5kdc[1180](info): commencing operation
krb5kdc: starting...
W20260925 02:26:46.919200 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:46 dist-test-slave-hj7s krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:47.033882 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34377
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:47.114805 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:47.115012 1193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:47.115036 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:47.116129 1193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:47.116166 1193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:47.116180 1193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:47.116194 1193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:47.116205 1193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:47.117620 1193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:34377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:47.117870 1193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:47.118050 1193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:47.122920 1199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:47.122982 1193 server_base.cc:1034] running on GCE node
W20260925 02:26:47.123030 1198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:47.122920 1201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:47.123409 1193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:47.123644 1193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:47.124799 1193 hybrid_clock.cc:648] HybridClock initialized: now 1790303207124794 us; error 33 us; skew 500 ppm
Sep 25 02:26:47 dist-test-slave-hj7s krb5kdc[1180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:47.127192 1193 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:47.127648 1193 webserver.cc:460] Webserver started at http://127.11.125.62:34593/ using document root <none> and password file <none>
I20260925 02:26:47.127880 1193 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:47.127960 1193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:47.128067 1193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:47.129110 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "67edbfb6393d4854bcd23dda420b658c"
format_stamp: "Formatted at 2026-09-25 02:26:47 on dist-test-slave-hj7s"
server_key: "f15e88fbcbc4e6e28a3622e5a74bf32b"
server_key_iv: "d0fd9e554b8711214a0ee98a91ab8949"
server_key_version: "encryptionkey@0"
I20260925 02:26:47.129420 1193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "67edbfb6393d4854bcd23dda420b658c"
format_stamp: "Formatted at 2026-09-25 02:26:47 on dist-test-slave-hj7s"
server_key: "f15e88fbcbc4e6e28a3622e5a74bf32b"
server_key_iv: "d0fd9e554b8711214a0ee98a91ab8949"
server_key_version: "encryptionkey@0"
I20260925 02:26:47.130740 1193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:47.131451 1208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:47.131639 1193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:47.131731 1193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "67edbfb6393d4854bcd23dda420b658c"
format_stamp: "Formatted at 2026-09-25 02:26:47 on dist-test-slave-hj7s"
server_key: "f15e88fbcbc4e6e28a3622e5a74bf32b"
server_key_iv: "d0fd9e554b8711214a0ee98a91ab8949"
server_key_version: "encryptionkey@0"
I20260925 02:26:47.131803 1193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:47.162918 1193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:47.164847 1193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:47.165040 1193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:47.169261 1193 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:34377
I20260925 02:26:47.169317 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:34377 every 8 connection(s)
I20260925 02:26:47.169673 1193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:47.170552 1261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:47.173028 1261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c: Bootstrap starting.
I20260925 02:26:47.173629 1261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:47.173874 1261 log.cc:824] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:47.174650 1261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c: No bootstrap required, opened a new log
I20260925 02:26:47.176252 1261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } }
I20260925 02:26:47.176409 1261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:47.176434 1261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67edbfb6393d4854bcd23dda420b658c, State: Initialized, Role: FOLLOWER
I20260925 02:26:47.176606 1261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } }
I20260925 02:26:47.176667 1261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:47.176715 1261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:47.176767 1261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:47.177706 1261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } }
I20260925 02:26:47.177821 1261 leader_election.cc:302] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67edbfb6393d4854bcd23dda420b658c; no voters:
I20260925 02:26:47.177987 1261 leader_election.cc:288] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:47.178030 1264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:47.178217 1264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [term 1 LEADER]: Becoming Leader. State: Replica: 67edbfb6393d4854bcd23dda420b658c, State: Running, Role: LEADER
I20260925 02:26:47.178297 1261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:47.178361 1264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } }
I20260925 02:26:47.178707 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1193
I20260925 02:26:47.178833 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:47.178809 1265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67edbfb6393d4854bcd23dda420b658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } } }
I20260925 02:26:47.178905 1265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:47.179050 11764 external_mini_cluster.cc:1428] Setting key db74a2d1e1eeccc8a01c08cf8d61d901
I20260925 02:26:47.179528 1266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67edbfb6393d4854bcd23dda420b658c. Latest consensus state: current_term: 1 leader_uuid: "67edbfb6393d4854bcd23dda420b658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67edbfb6393d4854bcd23dda420b658c" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34377 } } }
I20260925 02:26:47.179627 1266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:47.180022 1273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:47.180583 1273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:47.182317 1273 catalog_manager.cc:1347] Generated new cluster ID: 43b5f4138a9a4504b60ef74e0b4c32ea
I20260925 02:26:47.182379 1273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:47.185470 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:47.179780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33321 (local address 127.11.125.62:34377)
0925 02:26:47.179977 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:47.179988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:47.180141 (+ 153us) server_negotiation.cc:407] Connection header received
0925 02:26:47.180320 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:47.180329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:47.180405 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:47.180560 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:47.180767 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:47.182047 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:47.183180 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:47.183848 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:47.183970 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:47.184708 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:47.184713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:47.184713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:47.184956 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:47.184996 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:47.185011 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:47.185104 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:47.185194 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 02:26:47.185227 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260925 02:26:47.186440 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:47.191591 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:47.187976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60099 (local address 127.11.125.62:34377)
0925 02:26:47.188027 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:47.188029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:47.188223 (+ 194us) server_negotiation.cc:407] Connection header received
0925 02:26:47.188271 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:47.188272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:47.188294 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:47.188355 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:47.188526 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:47.189307 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:47.190276 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:47.190600 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:47.190667 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:47.191332 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:47.191333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:47.191334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:47.191338 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:47.191362 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:47.191366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:47.191432 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:47.191498 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:26:47.191511 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:47.200410 1273 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:47.200970 1273 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:47.206832 1273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67edbfb6393d4854bcd23dda420b658c: Generated new TSK 0
I20260925 02:26:47.207046 1273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:47.214807 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1193
W20260925 02:26:47.219445 1178 connection.cc:535] client connection to 127.11.125.62:34377 recv error: Network error: failed to read from TLS socket (remote: 127.11.125.62:34377): Connection reset by peer (error 104)
2026-09-25T02:26:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:47 dist-test-slave-hj7s krb5kdc[1292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:47 dist-test-slave-hj7s krb5kdc[1292](info): set up 2 sockets
Sep 25 02:26:47 dist-test-slave-hj7s krb5kdc[1292](info): commencing operation
krb5kdc: starting...
W20260925 02:26:48.291158 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:48 dist-test-slave-hj7s krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:48.412770 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36133
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:38861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:48.495911 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:48.496176 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:48.496219 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:48.497521 1305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:48.497591 1305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:48.497614 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:48.497637 1305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:48.497665 1305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:48.499236 1305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:38861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:36133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:48.499528 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:48.499773 1305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:48.504639 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:48.504639 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:48.504627 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:48.504882 1305 server_base.cc:1034] running on GCE node
I20260925 02:26:48.505143 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:48.505439 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:48.506601 1305 hybrid_clock.cc:648] HybridClock initialized: now 1790303208506594 us; error 36 us; skew 500 ppm
Sep 25 02:26:48 dist-test-slave-hj7s krb5kdc[1292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303208, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:48.509137 1305 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:48.509593 1305 webserver.cc:460] Webserver started at http://127.11.125.62:35207/ using document root <none> and password file <none>
I20260925 02:26:48.509857 1305 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:48.509928 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:48.510079 1305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:48.511183 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "6481bd2fbddb4e8681ec13c35eb07214"
format_stamp: "Formatted at 2026-09-25 02:26:48 on dist-test-slave-hj7s"
server_key: "3b107de972a401e546df4486871611e5"
server_key_iv: "2fe082319db455d8762083c3f2c5ebf2"
server_key_version: "encryptionkey@0"
I20260925 02:26:48.511585 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "6481bd2fbddb4e8681ec13c35eb07214"
format_stamp: "Formatted at 2026-09-25 02:26:48 on dist-test-slave-hj7s"
server_key: "3b107de972a401e546df4486871611e5"
server_key_iv: "2fe082319db455d8762083c3f2c5ebf2"
server_key_version: "encryptionkey@0"
I20260925 02:26:48.513019 1305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 02:26:48.513780 1320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:48.513949 1305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:48.514053 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "6481bd2fbddb4e8681ec13c35eb07214"
format_stamp: "Formatted at 2026-09-25 02:26:48 on dist-test-slave-hj7s"
server_key: "3b107de972a401e546df4486871611e5"
server_key_iv: "2fe082319db455d8762083c3f2c5ebf2"
server_key_version: "encryptionkey@0"
I20260925 02:26:48.514134 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:48.517647 1305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:48.519665 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:48.519836 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:48.523867 1305 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:36133
I20260925 02:26:48.523917 1372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:36133 every 8 connection(s)
I20260925 02:26:48.524269 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:48.525362 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:48.527694 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1305
I20260925 02:26:48.527705 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214: Bootstrap starting.
I20260925 02:26:48.527827 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:48.528008 11764 external_mini_cluster.cc:1428] Setting key 113a57c3588e2bcf6cf56eacad3c3bcf
I20260925 02:26:48.528460 1373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:48.528827 1373 log.cc:824] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:48.529666 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214: No bootstrap required, opened a new log
I20260925 02:26:48.530817 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:48.528520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46149 (local address 127.11.125.62:36133)
0925 02:26:48.528713 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:48.528721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:48.528751 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:26:48.529383 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:48.529391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:48.529419 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:48.529540 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:48.530355 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:48.530358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:48.530358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:48.530362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:48.530399 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:48.530409 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:48.530475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:48.530545 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:26:48.530593 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260925 02:26:48.531940 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:48.532227 1373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } }
I20260925 02:26:48.532366 1373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:48.532388 1373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6481bd2fbddb4e8681ec13c35eb07214, State: Initialized, Role: FOLLOWER
I20260925 02:26:48.532529 1373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } }
I20260925 02:26:48.532609 1373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:48.532665 1373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:48.532718 1373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:48.533452 1373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } }
I20260925 02:26:48.533571 1373 leader_election.cc:302] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6481bd2fbddb4e8681ec13c35eb07214; no voters:
I20260925 02:26:48.533731 1373 leader_election.cc:288] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:48.533756 1383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:48.533915 1383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [term 1 LEADER]: Becoming Leader. State: Replica: 6481bd2fbddb4e8681ec13c35eb07214, State: Running, Role: LEADER
I20260925 02:26:48.534041 1373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:48.534026 1383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } }
I20260925 02:26:48.534412 1383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6481bd2fbddb4e8681ec13c35eb07214. Latest consensus state: current_term: 1 leader_uuid: "6481bd2fbddb4e8681ec13c35eb07214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } } }
I20260925 02:26:48.534493 1383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:48.534593 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:48.533484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47947 (local address 127.11.125.62:36133)
0925 02:26:48.533526 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:48.533528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:48.533633 (+ 105us) server_negotiation.cc:407] Connection header received
0925 02:26:48.533704 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:48.533705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:48.533710 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:48.533759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:48.534350 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:48.534351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:48.534352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:48.534356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:48.534371 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:48.534376 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:48.534407 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:48.534520 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 02:26:48.534533 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:48.534854 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6481bd2fbddb4e8681ec13c35eb07214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6481bd2fbddb4e8681ec13c35eb07214" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 36133 } } }
I20260925 02:26:48.534976 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:48.535387 1389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:48.535884 1389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:48.537762 1389 catalog_manager.cc:1347] Generated new cluster ID: 18ed264632ea4c14816a149ac14ca441
I20260925 02:26:48.537815 1389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:48.549778 1389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:48.549968 1389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:48.558789 1389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6481bd2fbddb4e8681ec13c35eb07214: Generated new TSK 0
I20260925 02:26:48.558985 1389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:48.563939 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1305
2026-09-25T02:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:48 dist-test-slave-hj7s krb5kdc[1404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:48 dist-test-slave-hj7s krb5kdc[1404](info): set up 2 sockets
Sep 25 02:26:48 dist-test-slave-hj7s krb5kdc[1404](info): commencing operation
krb5kdc: starting...
W20260925 02:26:49.611294 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:49.732052 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44569
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:49.815282 1417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:49.815483 1417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:49.815506 1417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:49.816728 1417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:49.816769 1417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:49.816784 1417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:49.816797 1417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:49.816854 1417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:49.818312 1417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:49.818557 1417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:49.818792 1417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:49.823728 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:49.823786 1417 server_base.cc:1034] running on GCE node
W20260925 02:26:49.823736 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:49.823746 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:49.824211 1417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:49.824507 1417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:49.825671 1417 hybrid_clock.cc:648] HybridClock initialized: now 1790303209825667 us; error 36 us; skew 500 ppm
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303209, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:49.828032 1417 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:49.828512 1417 webserver.cc:460] Webserver started at http://127.11.125.62:43031/ using document root <none> and password file <none>
I20260925 02:26:49.828732 1417 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:49.828799 1417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:49.828975 1417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:49.830037 1417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "dba3d6f6b79c439d808eaeb02fc1ef65"
format_stamp: "Formatted at 2026-09-25 02:26:49 on dist-test-slave-hj7s"
server_key: "55d5618a57efa93b4d9562ebf6fb41d4"
server_key_iv: "bc10b14f7c083b0d538cdb29adabb048"
server_key_version: "encryptionkey@0"
I20260925 02:26:49.830420 1417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "dba3d6f6b79c439d808eaeb02fc1ef65"
format_stamp: "Formatted at 2026-09-25 02:26:49 on dist-test-slave-hj7s"
server_key: "55d5618a57efa93b4d9562ebf6fb41d4"
server_key_iv: "bc10b14f7c083b0d538cdb29adabb048"
server_key_version: "encryptionkey@0"
I20260925 02:26:49.831863 1417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:49.832624 1433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:49.832794 1417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:49.832897 1417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "dba3d6f6b79c439d808eaeb02fc1ef65"
format_stamp: "Formatted at 2026-09-25 02:26:49 on dist-test-slave-hj7s"
server_key: "55d5618a57efa93b4d9562ebf6fb41d4"
server_key_iv: "bc10b14f7c083b0d538cdb29adabb048"
server_key_version: "encryptionkey@0"
I20260925 02:26:49.833015 1417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:49.844262 1417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:49.846179 1417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:49.846347 1417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:49.850399 1417 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:44569
I20260925 02:26:49.850435 1486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:44569 every 8 connection(s)
I20260925 02:26:49.850814 1417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:49.851663 1487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:49.854075 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65: Bootstrap starting.
I20260925 02:26:49.854672 1487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:49.854969 1487 log.cc:824] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:49.855676 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65: No bootstrap required, opened a new log
I20260925 02:26:49.856817 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1417
I20260925 02:26:49.856966 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:49.857168 11764 external_mini_cluster.cc:1428] Setting key 7fff4ba07dc5831167bf48c1dcd16bfe
I20260925 02:26:49.857471 1487 raft_consensus.cc:348] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } }
I20260925 02:26:49.857707 1487 raft_consensus.cc:374] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:49.857811 1487 raft_consensus.cc:729] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba3d6f6b79c439d808eaeb02fc1ef65, State: Initialized, Role: FOLLOWER
I20260925 02:26:49.857992 1487 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } }
I20260925 02:26:49.858084 1487 raft_consensus.cc:388] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:49.858127 1487 raft_consensus.cc:482] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:49.858179 1487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:49.858937 1487 raft_consensus.cc:504] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } }
I20260925 02:26:49.859061 1487 leader_election.cc:302] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dba3d6f6b79c439d808eaeb02fc1ef65; no voters:
I20260925 02:26:49.859225 1487 leader_election.cc:288] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:49.859253 1492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:49.859364 1492 raft_consensus.cc:686] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [term 1 LEADER]: Becoming Leader. State: Replica: dba3d6f6b79c439d808eaeb02fc1ef65, State: Running, Role: LEADER
I20260925 02:26:49.859503 1487 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:49.859477 1492 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } }
I20260925 02:26:49.859843 1492 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } } }
I20260925 02:26:49.859994 1492 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:49.860183 1492 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba3d6f6b79c439d808eaeb02fc1ef65. Latest consensus state: current_term: 1 leader_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba3d6f6b79c439d808eaeb02fc1ef65" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 44569 } } }
I20260925 02:26:49.860262 1492 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:49.860618 1500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:49.861207 1500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:49.862871 1500 catalog_manager.cc:1347] Generated new cluster ID: 372525e676c7425db9b335a1bbab405a
I20260925 02:26:49.862926 1500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:49.868111 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:49.857716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56297 (local address 127.11.125.62:44569)
0925 02:26:49.857902 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:49.857912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:49.857949 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:26:49.858088 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:49.858093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:49.858164 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:49.858304 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:49.859178 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:49.860607 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:49.861745 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:49.862345 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:49.862452 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:49.864807 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:49.864811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:49.864812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:49.864823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:49.866573 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:49.866961 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:49.866964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:49.866965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:49.867069 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:49.867283 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:49.867284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:49.867285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:49.867535 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:49.867700 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:49.867806 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:26:49.867959 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 02:26:49.869082 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:49.877264 1500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:49.877980 1500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:49.879189 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:49.870433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47821 (local address 127.11.125.62:44569)
0925 02:26:49.870469 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:49.870471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:49.873098 (+ 2627us) server_negotiation.cc:407] Connection header received
0925 02:26:49.873169 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:49.873170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:49.873183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:49.873243 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:49.873791 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:49.874285 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:49.874999 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:49.875214 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:49.875288 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:49.876458 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:49.876460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:49.876460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:49.876467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:49.878093 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:49.878411 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:49.878412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:49.878412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:49.878466 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:49.878679 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:49.878680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:49.878680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:49.878787 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:49.878899 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:49.879007 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:26:49.879090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:49.883919 1500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dba3d6f6b79c439d808eaeb02fc1ef65: Generated new TSK 0
I20260925 02:26:49.884109 1500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:49.889207 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1417
2026-09-25T02:26:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1518](info): set up 2 sockets
Sep 25 02:26:49 dist-test-slave-hj7s krb5kdc[1518](info): commencing operation
krb5kdc: starting...
W20260925 02:26:50.931262 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:51 dist-test-slave-hj7s krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:51.051883 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34069
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:34069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:51.132591 1532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:51.132788 1532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:51.132812 1532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:51.134014 1532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:51.134063 1532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:51.134078 1532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:51.134091 1532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:51.134143 1532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:51.135608 1532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:34069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:34069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:51.135900 1532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:51.136125 1532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:51.141011 1540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:51.141047 1532 server_base.cc:1034] running on GCE node
W20260925 02:26:51.141019 1537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:51.141031 1538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:51.141412 1532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:51.141705 1532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:51.142869 1532 hybrid_clock.cc:648] HybridClock initialized: now 1790303211142925 us; error 102 us; skew 500 ppm
Sep 25 02:26:51 dist-test-slave-hj7s krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303211, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:51.153692 1532 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:51.154187 1532 webserver.cc:460] Webserver started at http://127.11.125.62:35229/ using document root <none> and password file <none>
I20260925 02:26:51.154417 1532 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:51.154487 1532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:51.154641 1532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:51.155720 1532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b"
format_stamp: "Formatted at 2026-09-25 02:26:51 on dist-test-slave-hj7s"
server_key: "4aa2a9cede1c22570933ce293fef7ced"
server_key_iv: "dc9f7ddfcf5a6fa70ce5e91c52e84b3d"
server_key_version: "encryptionkey@0"
I20260925 02:26:51.156100 1532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b"
format_stamp: "Formatted at 2026-09-25 02:26:51 on dist-test-slave-hj7s"
server_key: "4aa2a9cede1c22570933ce293fef7ced"
server_key_iv: "dc9f7ddfcf5a6fa70ce5e91c52e84b3d"
server_key_version: "encryptionkey@0"
I20260925 02:26:51.157512 1532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 02:26:51.158237 1547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:51.158440 1532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:51.158524 1532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b"
format_stamp: "Formatted at 2026-09-25 02:26:51 on dist-test-slave-hj7s"
server_key: "4aa2a9cede1c22570933ce293fef7ced"
server_key_iv: "dc9f7ddfcf5a6fa70ce5e91c52e84b3d"
server_key_version: "encryptionkey@0"
I20260925 02:26:51.158609 1532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:51.164813 1532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:51.166841 1532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:51.167012 1532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:51.172597 1532 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:34069
I20260925 02:26:51.172695 1599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:34069 every 8 connection(s)
I20260925 02:26:51.173096 1532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:51.176106 1600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:51.176373 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1532
I20260925 02:26:51.176509 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:51.176757 11764 external_mini_cluster.cc:1428] Setting key 608883e4f436087d2319e40315c556c7
I20260925 02:26:51.179390 1600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b: Bootstrap starting.
I20260925 02:26:51.180115 1600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:51.180399 1600 log.cc:824] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:51.183281 1600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b: No bootstrap required, opened a new log
I20260925 02:26:51.183992 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:51.177627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51897 (local address 127.11.125.62:34069)
0925 02:26:51.177843 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:51.177854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:51.177889 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:26:51.178451 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:51.178460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:51.178526 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:51.178675 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:51.179035 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:51.180428 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:51.181702 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:51.182113 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:51.182224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:51.183076 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:51.183081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:51.183081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:51.183281 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:51.183333 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:51.183350 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:51.183414 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:51.183686 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:26:51.183756 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260925 02:26:51.185225 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:51.186018 1600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } }
I20260925 02:26:51.186210 1600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:51.186257 1600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4ec34358e1434bae4c7bdc2c0dda5b, State: Initialized, Role: FOLLOWER
I20260925 02:26:51.186381 1600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } }
I20260925 02:26:51.186482 1600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:51.186529 1600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:51.186581 1600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:51.187282 1600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } }
I20260925 02:26:51.187419 1600 leader_election.cc:302] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f4ec34358e1434bae4c7bdc2c0dda5b; no voters:
I20260925 02:26:51.187588 1600 leader_election.cc:288] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:51.187634 1611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:51.187840 1611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [term 1 LEADER]: Becoming Leader. State: Replica: 7f4ec34358e1434bae4c7bdc2c0dda5b, State: Running, Role: LEADER
I20260925 02:26:51.187846 1600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:51.188014 1611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } }
I20260925 02:26:51.188402 1611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4ec34358e1434bae4c7bdc2c0dda5b. Latest consensus state: current_term: 1 leader_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } } }
I20260925 02:26:51.188410 1612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4ec34358e1434bae4c7bdc2c0dda5b" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 34069 } } }
I20260925 02:26:51.188512 1612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:51.188506 1611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:51.188738 1620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:51.189281 1620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:51.190907 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:51.186963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51491 (local address 127.11.125.62:34069)
0925 02:26:51.187001 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:51.187002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:51.187034 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:26:51.187110 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:51.187110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:51.187121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:51.187181 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:51.187349 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:51.188264 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:51.189471 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:51.189758 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:51.189855 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:51.190548 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:51.190549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:51.190549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:51.190554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:51.190569 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:51.190573 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:51.190786 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:51.190807 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 02:26:51.190829 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:51.193795 1620 catalog_manager.cc:1347] Generated new cluster ID: 82a0f5bb71584350b664cca0e3ef6ea6
I20260925 02:26:51.193856 1620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:51.215562 1620 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:51.216441 1620 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:51.224850 1620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f4ec34358e1434bae4c7bdc2c0dda5b: Generated new TSK 0
I20260925 02:26:51.225131 1620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:51.229190 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1532
2026-09-25T02:26:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:51 dist-test-slave-hj7s krb5kdc[1632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:51 dist-test-slave-hj7s krb5kdc[1632](info): set up 2 sockets
Sep 25 02:26:51 dist-test-slave-hj7s krb5kdc[1632](info): commencing operation
krb5kdc: starting...
W20260925 02:26:52.291301 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:52 dist-test-slave-hj7s krb5kdc[1632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:52.411952 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35763
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:35763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:52.492578 1645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:52.492823 1645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:52.492861 1645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:52.494032 1645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:52.494104 1645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:52.494133 1645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:52.494165 1645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:52.494189 1645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:52.495692 1645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:35763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:52.495961 1645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:52.496184 1645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:52.501008 1650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:52.501071 1653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:52.501094 1651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:52.501111 1645 server_base.cc:1034] running on GCE node
I20260925 02:26:52.501492 1645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:52.501724 1645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:52.502866 1645 hybrid_clock.cc:648] HybridClock initialized: now 1790303212502856 us; error 31 us; skew 500 ppm
Sep 25 02:26:52 dist-test-slave-hj7s krb5kdc[1632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303212, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:52.505395 1645 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:52.505836 1645 webserver.cc:460] Webserver started at http://127.11.125.62:34855/ using document root <none> and password file <none>
I20260925 02:26:52.506016 1645 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:52.506054 1645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:52.506141 1645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:52.507115 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "d505cde59bc347779e5944ba1603f5e2"
format_stamp: "Formatted at 2026-09-25 02:26:52 on dist-test-slave-hj7s"
server_key: "b8402cc301b6f5f4c94904a0b8dd36bc"
server_key_iv: "5a9f6dbc7b2304f645a8448483266b86"
server_key_version: "encryptionkey@0"
I20260925 02:26:52.507424 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "d505cde59bc347779e5944ba1603f5e2"
format_stamp: "Formatted at 2026-09-25 02:26:52 on dist-test-slave-hj7s"
server_key: "b8402cc301b6f5f4c94904a0b8dd36bc"
server_key_iv: "5a9f6dbc7b2304f645a8448483266b86"
server_key_version: "encryptionkey@0"
I20260925 02:26:52.508669 1645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:52.509438 1660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:52.509635 1645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 02:26:52.509745 1645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "d505cde59bc347779e5944ba1603f5e2"
format_stamp: "Formatted at 2026-09-25 02:26:52 on dist-test-slave-hj7s"
server_key: "b8402cc301b6f5f4c94904a0b8dd36bc"
server_key_iv: "5a9f6dbc7b2304f645a8448483266b86"
server_key_version: "encryptionkey@0"
I20260925 02:26:52.509824 1645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:52.513098 1645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:52.515393 1645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:52.515556 1645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:52.520148 1645 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:35763
I20260925 02:26:52.520200 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:35763 every 8 connection(s)
I20260925 02:26:52.520565 1645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:52.521600 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:52.523886 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2: Bootstrap starting.
I20260925 02:26:52.524443 1713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:52.524684 1713 log.cc:824] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:52.525396 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2: No bootstrap required, opened a new log
I20260925 02:26:52.526357 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1645
I20260925 02:26:52.526487 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:52.526693 11764 external_mini_cluster.cc:1428] Setting key 926a06e92b9cdfdee3632e8a92f71c96
I20260925 02:26:52.527307 1713 raft_consensus.cc:348] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } }
I20260925 02:26:52.527515 1713 raft_consensus.cc:374] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:26:52.527567 1713 raft_consensus.cc:729] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d505cde59bc347779e5944ba1603f5e2, State: Initialized, Role: FOLLOWER
I20260925 02:26:52.527698 1713 consensus_queue.cc:260] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } }
I20260925 02:26:52.527792 1713 raft_consensus.cc:388] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:26:52.527828 1713 raft_consensus.cc:482] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:26:52.527879 1713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:52.528580 1713 raft_consensus.cc:504] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } }
I20260925 02:26:52.528717 1713 leader_election.cc:302] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d505cde59bc347779e5944ba1603f5e2; no voters:
I20260925 02:26:52.528891 1713 leader_election.cc:288] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:26:52.528954 1718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:52.529062 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:52.527245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48301 (local address 127.11.125.62:35763)
0925 02:26:52.527432 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:52.527440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:52.527467 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:26:52.527595 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:52.527603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:52.527636 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:52.527762 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:52.528496 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:52.528499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:52.528499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:52.528503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:52.528541 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:52.528552 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:52.528608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:52.528725 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:26:52.528770 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260925 02:26:52.529071 1718 raft_consensus.cc:686] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [term 1 LEADER]: Becoming Leader. State: Replica: d505cde59bc347779e5944ba1603f5e2, State: Running, Role: LEADER
I20260925 02:26:52.529170 1713 sys_catalog.cc:564] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:52.529178 1718 consensus_queue.cc:237] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } }
I20260925 02:26:52.529510 1718 sys_catalog.cc:455] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d505cde59bc347779e5944ba1603f5e2. Latest consensus state: current_term: 1 leader_uuid: "d505cde59bc347779e5944ba1603f5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } } }
I20260925 02:26:52.529636 1718 sys_catalog.cc:458] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:52.529524 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d505cde59bc347779e5944ba1603f5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d505cde59bc347779e5944ba1603f5e2" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 35763 } } }
I20260925 02:26:52.529863 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:52.529881 1726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:52.529989 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:52.530435 1726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:52.531880 1740 mvcc.cc:205] Tried to move back new op lower bound from 7333081958529413120 to 7333081958520041472. Current Snapshot: MvccSnapshot[applied={T|T < 7333081958529413120}]
I20260925 02:26:52.532974 1726 catalog_manager.cc:1347] Generated new cluster ID: d8b98e01bff54f71a31319fe29fc3bfc
I20260925 02:26:52.533026 1726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:52.533469 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:52.532201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55477 (local address 127.11.125.62:35763)
0925 02:26:52.532234 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:52.532236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:52.532447 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:26:52.532520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:52.532521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:52.532525 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:52.532595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:52.533263 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:52.533264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:52.533264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:26:52.533268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:52.533283 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:26:52.533287 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:52.533340 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:52.533401 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:26:52.533413 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:26:52.538458 1726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:52.538619 1726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:52.542160 1726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d505cde59bc347779e5944ba1603f5e2: Generated new TSK 0
I20260925 02:26:52.542341 1726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:52.548082 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1645
2026-09-25T02:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1319 ms)
[----------] 18 tests from AuthTokenIssuingTest (24087 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:52 dist-test-slave-hj7s krb5kdc[1745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:52 dist-test-slave-hj7s krb5kdc[1745](info): set up 2 sockets
Sep 25 02:26:52 dist-test-slave-hj7s krb5kdc[1745](info): commencing operation
krb5kdc: starting...
W20260925 02:26:53.604434 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:53 dist-test-slave-hj7s krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:53.725905 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39811
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:53.808856 1759 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:53.809154 1759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:53.809171 1759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:53.809216 1759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:53.810375 1759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:53.810415 1759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:53.810429 1759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:53.810475 1759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:53.810503 1759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:53.812038 1759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:39811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:53.812296 1759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:53.812526 1759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:53.817513 1765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:53.817531 1767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:53.817531 1764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:53.817734 1759 server_base.cc:1034] running on GCE node
I20260925 02:26:53.818029 1759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:53.818305 1759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:53.819521 1759 hybrid_clock.cc:648] HybridClock initialized: now 1790303213819508 us; error 26 us; skew 500 ppm
Sep 25 02:26:53 dist-test-slave-hj7s krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:53.822047 1759 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:53.822439 1759 webserver.cc:460] Webserver started at http://127.11.125.62:37617/ using document root <none> and password file <none>
I20260925 02:26:53.822613 1759 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:53.822654 1759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:53.822741 1759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:53.823728 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "0f178f71c0754a00b129c3a932f7e1b6"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "e79be51a878b1c96c29889f702cc46ac"
server_key_iv: "257fe51543af173958429a4dd6fee525"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.824023 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "0f178f71c0754a00b129c3a932f7e1b6"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "e79be51a878b1c96c29889f702cc46ac"
server_key_iv: "257fe51543af173958429a4dd6fee525"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.825369 1759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:53.826056 1774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:53.826236 1759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:53.826326 1759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "0f178f71c0754a00b129c3a932f7e1b6"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "e79be51a878b1c96c29889f702cc46ac"
server_key_iv: "257fe51543af173958429a4dd6fee525"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.826385 1759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:53.835374 1759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:53.837738 1759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:53.837867 1759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:53.841979 1759 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:39811
I20260925 02:26:53.842012 1827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:39811 every 8 connection(s)
I20260925 02:26:53.842384 1759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:53.843451 1828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:53.845139 1828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:53.850363 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1759
I20260925 02:26:53.850462 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:53.850660 11764 external_mini_cluster.cc:1428] Setting key cdb1cf30ada136bce8b2a3dd28e66c86
Sep 25 02:26:53 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:53.855027 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.845877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39811 (local address 127.0.0.1:37016)
0925 02:26:53.846059 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.846099 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:53.846579 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:53.846837 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:53.846840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:53.847081 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:53.847242 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:53.847246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:53.848214 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:53.848215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:53.848882 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:53.848883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:53.849413 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:53.849414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:53.849469 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:53.850016 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:53.850025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:53.851661 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:53.853514 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:53.853520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:53.853522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:53.853745 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:53.854000 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:53.854001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:53.854001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:53.854085 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:53.854446 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:53.854451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:53.854519 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 02:26:53.854613 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:26:53.854855 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":93,"thread_start_us":89,"threads_started":1}
I20260925 02:26:53.855031 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.846062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.11.125.62:39811)
0925 02:26:53.846186 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:53.846189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:53.846219 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:26:53.846706 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:53.846708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:53.846731 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:53.846808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:53.847322 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:53.848131 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:53.849013 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:53.849320 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:53.849399 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:53.851745 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:53.851754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:53.851754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:53.851762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:53.853438 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:53.853841 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:53.853844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:53.853845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:53.853930 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:53.854152 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:53.854153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:53.854154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:53.854366 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:53.854455 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:53.854646 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:26:53.854782 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260925 02:26:53.855741 1828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:53.856089 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.855968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59714)
0925 02:26:53.856016 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.856060 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 02:26:53.856218 1778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
I20260925 02:26:53.856683 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.856588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59716)
0925 02:26:53.856637 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.856661 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:26:53.856789 1828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:33713: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.11.125.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
I20260925 02:26:53.876008 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:33713
--webserver_interface=127.11.125.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:53.906009 1828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } attempt: 1
I20260925 02:26:53.906577 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.906434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59732)
0925 02:26:53.906504 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.906548 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:26:53.906965 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.906887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59740)
0925 02:26:53.906925 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.906950 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 02:26:53.907071 1828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:33713: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
W20260925 02:26:53.962460 1835 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:53.962661 1835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:53.962679 1835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:53.962699 1835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:53.963860 1835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:53.963900 1835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:53.963914 1835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:53.963953 1835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:53.963986 1835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:53.965478 1835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.125.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:53.965731 1835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:53.965969 1835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:53.970906 1840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:53.970924 1841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:53.970906 1843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:53.971140 1835 server_base.cc:1034] running on GCE node
I20260925 02:26:53.971459 1835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:53.971755 1835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:53.972955 1835 hybrid_clock.cc:648] HybridClock initialized: now 1790303213972932 us; error 36 us; skew 500 ppm
Sep 25 02:26:53 dist-test-slave-hj7s krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:53.975216 1828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } attempt: 2
I20260925 02:26:53.975606 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.975480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59752)
0925 02:26:53.975538 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.975569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:26:53.975692 1835 init.cc:377] Logged in from keytab as kudu/127.11.125.61@KRBTEST.COM (short username kudu)
I20260925 02:26:53.975910 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:53.975822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59756)
0925 02:26:53.975866 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:53.975890 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 02:26:53.976052 1828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:33713: Network error: Client connection negotiation failed: client connection to 127.11.125.61:33713: connect: Connection refused (error 111)
I20260925 02:26:53.976289 1835 webserver.cc:460] Webserver started at http://127.11.125.61:39939/ using document root <none> and password file <none>
I20260925 02:26:53.976498 1835 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:53.976536 1835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:53.976629 1835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:53.977739 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data/instance:
uuid: "c63fc08037c24fb1974e0a0d1d4573e2"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "3ab2ec13515c3b5aae776c4a3e824814"
server_key_iv: "c1dc9e00c7506d8b33d425d94fd1717b"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.978044 1835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal/instance:
uuid: "c63fc08037c24fb1974e0a0d1d4573e2"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "3ab2ec13515c3b5aae776c4a3e824814"
server_key_iv: "c1dc9e00c7506d8b33d425d94fd1717b"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.979351 1835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:26:53.980042 1850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:53.980221 1835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:53.980304 1835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "c63fc08037c24fb1974e0a0d1d4573e2"
format_stamp: "Formatted at 2026-09-25 02:26:53 on dist-test-slave-hj7s"
server_key: "3ab2ec13515c3b5aae776c4a3e824814"
server_key_iv: "c1dc9e00c7506d8b33d425d94fd1717b"
server_key_version: "encryptionkey@0"
I20260925 02:26:53.980358 1835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:53.992379 1835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:53.994663 1835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:53.994776 1835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:53.998942 1835 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.61:33713
I20260925 02:26:53.999018 1903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.61:33713 every 8 connection(s)
I20260925 02:26:53.999366 1835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
I20260925 02:26:54.000332 1904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:54.001116 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1835
I20260925 02:26:54.001251 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-1/wal/instance
I20260925 02:26:54.001432 11764 external_mini_cluster.cc:1428] Setting key 1098c6397b761170845d466014a8623e
I20260925 02:26:54.002146 1904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.60@KRBTEST.COM; defaulting to no policy
I20260925 02:26:54.011422 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.002855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.11.125.62:39811)
0925 02:26:54.002918 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.002920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.003166 (+ 246us) server_negotiation.cc:407] Connection header received
0925 02:26:54.003618 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.003619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.003638 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.003727 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.004283 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.005122 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.006116 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.006480 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.006566 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.008658 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.008660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.008661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.008670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.010186 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.010575 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.010575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.010576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.010627 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.010840 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.010840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.010841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.010945 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.011021 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.011282 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.011356 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.60@KRBTEST.COM" created.
I20260925 02:26:54.011663 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.002856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39811 (local address 127.0.0.1:37028)
0925 02:26:54.003029 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.003072 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.003526 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.003770 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.003774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.004008 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.004192 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.004196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.005222 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.005223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.006021 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.006023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.006581 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.006582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.006697 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.007197 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.007206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.008548 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.010268 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.010276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.010278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.010506 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.010683 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.010684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.010684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.010785 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.011031 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.011036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.011166 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.011253 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.011398 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260925 02:26:54.012228 1904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:54.020486 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.012534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59758)
0925 02:26:54.012594 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.012607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.012654 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.012877 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.012878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.013071 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.013131 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.013133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.014455 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.014456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.014944 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.014945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.015641 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.015642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.015696 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.016092 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.016097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.017633 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.019411 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.019412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.019412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.019631 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.019863 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.019864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.019864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.019920 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.020218 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.020219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.020267 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.020333 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.020416 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:26:54.020634 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.012573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.11.125.61:33713)
0925 02:26:54.012725 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.012729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.012748 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:26:54.012776 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.012778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.012788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.012854 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.013217 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.014361 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.015030 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.015561 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.015647 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.017728 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.017736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.017736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.017742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.019321 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.019721 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.019725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.019725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.019785 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.019976 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.019977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.019977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.020164 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.020232 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.020359 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.020467 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260925 02:26:54.021250 1904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:54.021672 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.021528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45842)
0925 02:26:54.021589 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.021647 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:26:54.021836 1851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
I20260925 02:26:54.022315 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.022225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45844)
0925 02:26:54.022274 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.022299 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 02:26:54.022439 1904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:43507: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
I20260925 02:26:54.027056 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:43507
--webserver_interface=127.11.125.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:54.067214 1828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } attempt: 3
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:54.071723 1904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } attempt: 1
I20260925 02:26:54.072136 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.072005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45854)
0925 02:26:54.072063 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.072096 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:26:54.072533 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.072412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45862)
0925 02:26:54.072475 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.072506 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:26:54.072634 1904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:43507: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
I20260925 02:26:54.074815 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.067603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59768)
0925 02:26:54.067675 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.067689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.067748 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.067907 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.067908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.068080 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.068149 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.068152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.068874 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.068875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.069445 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.069447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.069863 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.069864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.069954 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.070472 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.070476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.071997 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.073884 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.073885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.073885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.074053 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.074259 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.074260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.074260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.074312 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.074580 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.074580 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.074622 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.074675 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.074752 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:26:54.074892 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.067681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.11.125.61:33713)
0925 02:26:54.067758 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.067760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.067787 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:26:54.067818 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.067821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.067837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.067900 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.068244 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.068767 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.069545 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.069781 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.069863 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.072096 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.072097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.072097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.072104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.073787 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.074126 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.074127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.074127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.074193 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.074372 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.074373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.074373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.074509 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.074575 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.074719 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.074801 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:26:54.075304 1828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:54.075647 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.075536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45866)
0925 02:26:54.075587 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.075613 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 02:26:54.075989 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.075933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45872)
0925 02:26:54.075955 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.075975 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 02:26:54.076089 1828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:43507: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
I20260925 02:26:54.107237 1828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } attempt: 1
I20260925 02:26:54.107777 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.107639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45878)
0925 02:26:54.107709 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.107749 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:26:54.108146 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.108062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45884)
0925 02:26:54.108107 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.108131 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 02:26:54.108273 1828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:43507: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
W20260925 02:26:54.112125 1911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:54.112323 1911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:54.112383 1911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:54.112414 1911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:54.113615 1911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:54.113655 1911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:54.113669 1911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:54.113682 1911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:54.113729 1911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:54.115190 1911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:39811,127.11.125.61:33713,127.11.125.60:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.125.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:54.115446 1911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:54.115679 1911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:54.120599 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:54.120618 1919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:54.120618 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:54.120657 1911 server_base.cc:1034] running on GCE node
I20260925 02:26:54.121064 1911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:54.121307 1911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:54.122468 1911 hybrid_clock.cc:648] HybridClock initialized: now 1790303214122451 us; error 42 us; skew 500 ppm
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:54.124836 1911 init.cc:377] Logged in from keytab as kudu/127.11.125.60@KRBTEST.COM (short username kudu)
I20260925 02:26:54.125334 1911 webserver.cc:460] Webserver started at http://127.11.125.60:42717/ using document root <none> and password file <none>
I20260925 02:26:54.125530 1911 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:54.125571 1911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:54.125659 1911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:54.126689 1911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data/instance:
uuid: "169564000f6f428ebc10b6f5974031d8"
format_stamp: "Formatted at 2026-09-25 02:26:54 on dist-test-slave-hj7s"
server_key: "baf39c704c28f2cfe1906ff2f17a4651"
server_key_iv: "0f28f2562c9a520bcb7ed73842fc39ce"
server_key_version: "encryptionkey@0"
I20260925 02:26:54.126986 1911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal/instance:
uuid: "169564000f6f428ebc10b6f5974031d8"
format_stamp: "Formatted at 2026-09-25 02:26:54 on dist-test-slave-hj7s"
server_key: "baf39c704c28f2cfe1906ff2f17a4651"
server_key_iv: "0f28f2562c9a520bcb7ed73842fc39ce"
server_key_version: "encryptionkey@0"
I20260925 02:26:54.128291 1911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 02:26:54.129065 1926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:54.129273 1911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:26:54.129388 1911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "169564000f6f428ebc10b6f5974031d8"
format_stamp: "Formatted at 2026-09-25 02:26:54 on dist-test-slave-hj7s"
server_key: "baf39c704c28f2cfe1906ff2f17a4651"
server_key_iv: "0f28f2562c9a520bcb7ed73842fc39ce"
server_key_version: "encryptionkey@0"
I20260925 02:26:54.129484 1911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:54.140797 1904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } attempt: 2
I20260925 02:26:54.140952 1911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:54.141326 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.141166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45900)
0925 02:26:54.141237 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.141279 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:26:54.141765 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.141653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45912)
0925 02:26:54.141710 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.141741 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:26:54.141921 1904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:43507: Network error: Client connection negotiation failed: client connection to 127.11.125.60:43507: connect: Connection refused (error 111)
I20260925 02:26:54.143244 1911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:54.143400 1911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:54.147807 1911 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.60:43507
I20260925 02:26:54.147868 1979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.60:43507 every 8 connection(s)
I20260925 02:26:54.148243 1911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
I20260925 02:26:54.149392 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:54.151010 1980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:54.152180 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 1911
I20260925 02:26:54.152282 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790302942477976-11764-0/minicluster-data/master-2/wal/instance
I20260925 02:26:54.152454 11764 external_mini_cluster.cc:1428] Setting key 90d9b65a6602d8e5cbba45d8db506c7b
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:54.162087 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.151732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:39811 (local address 127.0.0.1:37036)
0925 02:26:54.151953 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.152009 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.152592 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.152945 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.152950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.153356 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.153591 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.153596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.154404 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.154405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.155556 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.155559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.156043 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.156044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.156181 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.156808 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.156817 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.158371 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.160358 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.160365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.160367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.160661 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.160940 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.160941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.160942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.161108 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.161403 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.161409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.161589 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.161689 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.161806 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":212,"thread_start_us":114,"threads_started":1}
I20260925 02:26:54.162273 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.152889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.11.125.60:43507)
0925 02:26:54.153123 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.153128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.153182 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:26:54.153233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.153235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.153260 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.153355 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.154054 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.155123 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.156011 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.156435 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.156498 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.159017 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.159026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.159027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.159034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.160877 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.161312 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.161316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.161316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.161399 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.161604 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.161605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.161605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.161787 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.161943 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.162043 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.162137 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":139,"threads_started":1}
I20260925 02:26:54.162652 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.151781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.11.125.62:39811)
0925 02:26:54.151878 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.151880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.152101 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:26:54.152725 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.152726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.152738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.152814 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.153681 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.154318 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.155637 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.155932 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.156005 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.158457 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.158458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.158459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.158466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.160264 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.160738 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.160739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.160740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.160805 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.161180 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.161181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.161181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.161329 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.161406 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.162128 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.162566 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:26:54.163096 1980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:54.171443 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.163407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.11.125.61:33713)
0925 02:26:54.163463 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.163466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.163474 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:26:54.163499 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.163500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.163510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.163560 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.163811 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.164547 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.165603 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.165926 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.165988 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.168596 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.168598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.168598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.168607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.170308 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170630 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.170632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.170632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.170691 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170906 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.170907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.170908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.171024 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.171095 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.171263 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.171339 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:26:54.171443 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.163330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:33713 (local address 127.0.0.1:59772)
0925 02:26:54.163380 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.163389 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.163426 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.163567 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.163567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.163703 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.163735 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.163737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.164651 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.164652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.165509 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.165512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.165999 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.166000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.166064 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.166616 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.166621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.168500 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.170399 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.170400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.170550 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.170769 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.170770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.170831 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.171098 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.171099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.171149 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.171221 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.171331 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:26:54.171505 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.163135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.11.125.62:39811)
0925 02:26:54.163170 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.163172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.163224 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:26:54.163295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.163296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.163306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.163369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.163846 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.164585 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.165603 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.165903 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.165983 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.168682 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.168683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.168684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.168689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.170210 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170584 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.170585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.170585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.170642 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.170869 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.170870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.170871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.170990 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.171176 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.171214 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.171292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:26:54.171952 1980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:54.178936 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.172232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45928)
0925 02:26:54.172263 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.172273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.172311 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.172446 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.172446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.172602 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.172640 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.172641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.173457 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.173458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.174012 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.174014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.174482 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.174483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.174536 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.174974 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.174978 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.176059 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.177907 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.177908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.177908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.178067 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.178277 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.178278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.178278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.178328 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.178601 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.178602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.178639 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.178725 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.178822 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 02:26:54.178949 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.172266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.11.125.60:43507)
0925 02:26:54.172312 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.172314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.172323 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:54.172368 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.172369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.172382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.172441 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.172714 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.173386 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.174087 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.174408 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.174467 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.176152 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.176153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.176153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.176159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.177807 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.178144 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.178145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.178146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.178206 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.178394 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.178395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.178395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.178535 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.178598 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.178748 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.178870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":22}
I20260925 02:26:54.179672 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.172372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.11.125.61:33713)
0925 02:26:54.172405 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.172407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.172418 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:54.172477 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.172478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.172488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.172538 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.173099 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.173621 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.174349 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.174585 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.174657 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.176999 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.177000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.177000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.177006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.178433 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.178740 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.178741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.178742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.178800 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.179224 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.179225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.179226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.179348 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.179478 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.179551 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.179610 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:26:54.180604 1980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: Bootstrap starting.
I20260925 02:26:54.181604 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:54.181682 1980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:54.181989 1980 log.cc:824] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:54.182881 1980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: No bootstrap required, opened a new log
I20260925 02:26:54.183451 1828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } attempt: 2
I20260925 02:26:54.182713 1802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0f178f71c0754a00b129c3a932f7e1b6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37042
I20260925 02:26:54.184540 1980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.184684 1980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169564000f6f428ebc10b6f5974031d8, State: Initialized, Role: FOLLOWER
I20260925 02:26:54.184780 1980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.185113 1980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:54.185096 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.185222 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:26:54.187052 1997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:54.187420 1997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:54.188019 1911 master_runner.cc:186] Error getting master registration for 127.11.125.62:39811: OK, instance_id { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" instance_seqno: 1790303213840782 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:26:54.190459 1911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:26:54.192763 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.183697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45940)
0925 02:26:54.183745 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.183757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.183796 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.185307 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.185308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.185488 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.185534 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.185536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.186601 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.186603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.187125 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.187126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.187614 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.187615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.187682 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.188151 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.188157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.189718 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.191632 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.191633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.191633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.191865 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.192072 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.192074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.192074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.192146 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.192445 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.192446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.192496 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.192576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.192673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 02:26:54.192771 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.183771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.11.125.60:43507)
0925 02:26:54.185160 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.185163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.185172 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:54.185193 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.185194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.185204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.185616 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.185649 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.186451 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.187251 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.187534 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.187618 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.189827 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.189829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.189829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.189836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.191544 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.191942 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.191943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.191944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.192001 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.192216 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.192217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.192217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.192370 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.192434 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.192609 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.192682 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1351}
I20260925 02:26:54.194355 1828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: Bootstrap starting.
I20260925 02:26:54.195102 1828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:54.195423 1828 log.cc:824] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:54.196305 1828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: No bootstrap required, opened a new log
I20260925 02:26:54.197726 1828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.197907 1828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f178f71c0754a00b129c3a932f7e1b6, State: Initialized, Role: FOLLOWER
I20260925 02:26:54.198002 1828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.198328 1828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:54.198388 1802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:26:54.198319 2000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.198449 1802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:54.199249 1802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.199462 1802 leader_election.cc:288] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [CANDIDATE]: Term 1 election: Requested vote from peers c63fc08037c24fb1974e0a0d1d4573e2 (127.11.125.61:33713), 169564000f6f428ebc10b6f5974031d8 (127.11.125.60:43507)
I20260925 02:26:54.200762 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f178f71c0754a00b129c3a932f7e1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "169564000f6f428ebc10b6f5974031d8"
I20260925 02:26:54.200949 1954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:26:54.201112 2011 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:54.201210 2011 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:54.201910 1954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f178f71c0754a00b129c3a932f7e1b6 in term 1.
I20260925 02:26:54.202175 1775 leader_election.cc:302] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f178f71c0754a00b129c3a932f7e1b6, 169564000f6f428ebc10b6f5974031d8; no voters:
I20260925 02:26:54.202382 2000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.202414 2017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:54.202669 2017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0f178f71c0754a00b129c3a932f7e1b6, State: Running, Role: LEADER
I20260925 02:26:54.202872 2017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.200966 1759 master_runner.cc:186] Error getting master registration for 127.11.125.61:33713: OK, instance_id { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" instance_seqno: 1790303213997754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:26:54.203187 1759 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:26:54.204207 2000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f178f71c0754a00b129c3a932f7e1b6. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.204309 2000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:54.212253 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.202649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.11.125.62:39811)
0925 02:26:54.202690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.202692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.203907 (+ 1215us) server_negotiation.cc:407] Connection header received
0925 02:26:54.204009 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.204010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.204021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.204050 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.204845 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.205583 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.206624 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.206911 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.206986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.208507 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.208508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.208509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.208516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.210408 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.211019 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.211020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.211020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.211092 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.211419 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.211420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.211421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.211569 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.211659 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.212065 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.212161 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:54.214465 2022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:54.214680 2022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:54.216542 1954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [term 1 FOLLOWER]: Refusing update from remote peer 0f178f71c0754a00b129c3a932f7e1b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:54.216941 2017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:54.218576 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f178f71c0754a00b129c3a932f7e1b6. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.218674 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.222028 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.203598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.11.125.60:43507)
0925 02:26:54.203654 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.203656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.209175 (+ 5519us) server_negotiation.cc:407] Connection header received
0925 02:26:54.209272 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.209273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.209284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.209313 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.210654 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.211401 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.215481 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.215777 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.215818 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.217888 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.217889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.217890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.217898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.219563 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.220859 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.220861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.220861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.220957 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.221424 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.221425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.221426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.221567 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.221650 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.221856 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.221936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:26:54.224043 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.203118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.11.125.61:33713)
0925 02:26:54.203173 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.203175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.209087 (+ 5912us) server_negotiation.cc:407] Connection header received
0925 02:26:54.210004 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.210006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.210019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.210050 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.212441 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.213210 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.214370 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.214668 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.214735 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.220327 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.220330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.220330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.220338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.221992 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.222435 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.222437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.222437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.222500 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.223107 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.223108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.223108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.223249 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.223350 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.223865 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.223950 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 02:26:54.224315 2017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.224416 2017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:54.224761 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.224852 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.225320 2022 catalog_manager.cc:1347] Generated new cluster ID: b057da0aef224d3c9e9d11dce9f7aead
I20260925 02:26:54.225373 2022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:26:54.229313 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333081965428183040 to 7333081965377859584. Current Snapshot: MvccSnapshot[applied={T|T < 7333081965428183040}]
I20260925 02:26:54.233067 1904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } attempt: 3
Sep 25 02:26:54 dist-test-slave-hj7s krb5kdc[1745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303213, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:54.240783 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.233331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:43507 (local address 127.0.0.1:45954)
0925 02:26:54.233395 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:54.233407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:54.233456 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:54.233587 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:54.233588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:54.233753 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:54.233802 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.233804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.235021 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.235022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.235795 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:54.235797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.236251 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.236252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:54.236298 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.236741 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:54.236746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:54.238151 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:54.239847 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.239848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.239848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.240039 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.240208 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:54.240209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:54.240209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:54.240258 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:54.240507 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:54.240508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:54.240547 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:26:54.240601 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:26:54.240686 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 02:26:54.241114 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.233426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.11.125.60:43507)
0925 02:26:54.233460 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.233462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.233470 (+ 8us) server_negotiation.cc:407] Connection header received
0925 02:26:54.233510 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.233511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.233520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.233570 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.233872 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.234543 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.235879 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.236159 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.236207 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.238238 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:54.238239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:54.238239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:54.238246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:54.239761 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.240103 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.240104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.240105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.240149 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:54.240312 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:54.240313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:54.240313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:54.240437 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:54.240485 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.240968 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.241035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 02:26:54.242581 1904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: Bootstrap starting.
I20260925 02:26:54.243378 1904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:54.243717 1904 log.cc:824] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:54.244467 2022 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:54.244793 1904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: No bootstrap required, opened a new log
I20260925 02:26:54.245220 2022 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:54.249097 1904 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.249414 1904 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63fc08037c24fb1974e0a0d1d4573e2, State: Initialized, Role: FOLLOWER
I20260925 02:26:54.249621 1904 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } }
I20260925 02:26:54.250368 1876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:54.251276 1876 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [term 1 FOLLOWER]: Refusing update from remote peer 0f178f71c0754a00b129c3a932f7e1b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:54.251590 2017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:54.250391 1877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f178f71c0754a00b129c3a932f7e1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c63fc08037c24fb1974e0a0d1d4573e2"
I20260925 02:26:54.252436 1877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f178f71c0754a00b129c3a932f7e1b6 in term 1.
I20260925 02:26:54.250268 1904 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:54.257936 2022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6: Generated new TSK 0
I20260925 02:26:54.258137 2022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:26:54.261011 2029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.261126 2029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.261173 2029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.261220 2029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.261050 2030 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f178f71c0754a00b129c3a932f7e1b6. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.261266 2030 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:54.261725 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333081965539528704 to 7333081965377859584. Current Snapshot: MvccSnapshot[applied={T|T < 7333081965428183040}]
I20260925 02:26:54.262517 2000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f178f71c0754a00b129c3a932f7e1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f178f71c0754a00b129c3a932f7e1b6" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 39811 } } peers { permanent_uuid: "c63fc08037c24fb1974e0a0d1d4573e2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 33713 } } peers { permanent_uuid: "169564000f6f428ebc10b6f5974031d8" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 43507 } } }
I20260925 02:26:54.262609 2000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f178f71c0754a00b129c3a932f7e1b6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:54.263356 2045 catalog_manager.cc:1259] Loaded cluster ID: b057da0aef224d3c9e9d11dce9f7aead
I20260925 02:26:54.263404 2045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: loading cluster ID for follower catalog manager: success
I20260925 02:26:54.264674 2045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: acquiring CA information for follower catalog manager: success
I20260925 02:26:54.266367 2045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c63fc08037c24fb1974e0a0d1d4573e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:26:54.286127 1831 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.284340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.11.125.62:39811)
0925 02:26:54.284389 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.284391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.285397 (+ 1006us) server_negotiation.cc:407] Connection header received
0925 02:26:54.285536 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.285537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.285550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.285580 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.286059 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37046: BlockingRecv error: recv got EOF from 127.0.0.1:37046 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 02:26:54.287003 1908 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.284703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.11.125.61:33713)
0925 02:26:54.284741 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.284743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.286439 (+ 1696us) server_negotiation.cc:407] Connection header received
0925 02:26:54.286588 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.286589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.286601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.286631 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.286948 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59794: BlockingRecv error: recv got EOF from 127.0.0.1:59794 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:26:54.287746 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.285153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.11.125.60:43507)
0925 02:26:54.285213 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.285215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.287256 (+ 2041us) server_negotiation.cc:407] Connection header received
0925 02:26:54.287367 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.287368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.287378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.287406 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.287689 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45968: BlockingRecv error: recv got EOF from 127.0.0.1:45968 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:26:54.292853 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.289014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.11.125.61:33713)
0925 02:26:54.289053 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.289055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.289643 (+ 588us) server_negotiation.cc:407] Connection header received
0925 02:26:54.289743 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.289744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.289780 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.289828 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.290051 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.290801 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.291947 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.292229 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.292268 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.292486 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.292601 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.292650 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.292761 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.292778 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:26:54.296171 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.295391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.11.125.60:43507)
0925 02:26:54.295428 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.295429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.295610 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:26:54.295718 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.295719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.295736 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.295768 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.296121 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45974: BlockingRecv error: recv got EOF from 127.0.0.1:45974 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:54.301915 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.298653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.11.125.61:33713)
0925 02:26:54.298688 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.298690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.298942 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:26:54.299041 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.299042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.299052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.299082 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.299259 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.299998 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.301150 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.301437 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.301477 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.301661 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.301693 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.301719 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.301820 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.301848 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 02:26:54.305104 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.304311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.11.125.60:43507)
0925 02:26:54.304372 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.304373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.304516 (+ 143us) server_negotiation.cc:407] Connection header received
0925 02:26:54.304622 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.304623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.304635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.304663 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.305051 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45988: BlockingRecv error: recv got EOF from 127.0.0.1:45988 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:26:54.311573 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.308229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.11.125.61:33713)
0925 02:26:54.308268 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.308270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.308518 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:26:54.308618 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.308619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.308629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.308657 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.308827 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.309645 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.310741 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.311026 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.311073 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.311282 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.311315 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.311361 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.311478 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.311500 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:26:54.314725 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.313899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.11.125.60:43507)
0925 02:26:54.313940 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.313942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.314136 (+ 194us) server_negotiation.cc:407] Connection header received
0925 02:26:54.314247 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.314248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.314260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.314291 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.314674 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46002: BlockingRecv error: recv got EOF from 127.0.0.1:46002 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 02:26:54.323231 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.319948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.11.125.61:33713)
0925 02:26:54.319987 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.319989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.320246 (+ 257us) server_negotiation.cc:407] Connection header received
0925 02:26:54.320350 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.320351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.320361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.320391 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.320570 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.321310 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.322428 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.322697 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.322743 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.322967 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.323002 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.323031 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.323138 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.323158 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:26:54.326699 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.325744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.11.125.60:43507)
0925 02:26:54.325783 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.325785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.326101 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:26:54.326204 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.326205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.326218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.326266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.326637 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46018: BlockingRecv error: recv got EOF from 127.0.0.1:46018 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:54.339529 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.336097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.11.125.61:33713)
0925 02:26:54.336143 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.336145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.336449 (+ 304us) server_negotiation.cc:407] Connection header received
0925 02:26:54.336559 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.336560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.336571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.336602 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.336794 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.337577 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.338698 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.338998 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.339040 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.339233 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.339268 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.339295 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.339400 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.339448 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:26:54.342859 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.342058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.11.125.60:43507)
0925 02:26:54.342102 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.342104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.342285 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:26:54.342384 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.342385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.342397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.342426 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.342789 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46032: BlockingRecv error: recv got EOF from 127.0.0.1:46032 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:54.363847 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.360329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.11.125.61:33713)
0925 02:26:54.360378 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.360380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.360711 (+ 331us) server_negotiation.cc:407] Connection header received
0925 02:26:54.360819 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.360820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.360843 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.360874 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.361112 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.361808 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.362970 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.363253 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.363299 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.363513 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.363547 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.363574 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.363727 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.363749 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 02:26:54.367436 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.366554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.11.125.60:43507)
0925 02:26:54.366598 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.366600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.366805 (+ 205us) server_negotiation.cc:407] Connection header received
0925 02:26:54.366926 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.366928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.366940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.366971 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.367357 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46040: BlockingRecv error: recv got EOF from 127.0.0.1:46040 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:54.403471 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.400872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.11.125.61:33713)
0925 02:26:54.400945 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.400948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.401130 (+ 182us) server_negotiation.cc:407] Connection header received
0925 02:26:54.401202 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.401204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.401215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.401292 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.401458 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.401957 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.402741 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.402942 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.403016 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.403177 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.403215 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.403276 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.403348 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.403370 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:26:54.407545 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.405261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.11.125.60:43507)
0925 02:26:54.405310 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.405311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.407006 (+ 1695us) server_negotiation.cc:407] Connection header received
0925 02:26:54.407069 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.407070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.407081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.407128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.407486 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46042: BlockingRecv error: recv got EOF from 127.0.0.1:46042 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:54.474941 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.472758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.11.125.61:33713)
0925 02:26:54.472830 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.472837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.472926 (+ 89us) server_negotiation.cc:407] Connection header received
0925 02:26:54.472980 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.472981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.472991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.473053 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.473178 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.473655 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.474368 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.474558 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.474608 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.474736 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.474763 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.474801 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.474863 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.474879 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:26:54.477854 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.476941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.11.125.60:43507)
0925 02:26:54.476988 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.476990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.477059 (+ 69us) server_negotiation.cc:407] Connection header received
0925 02:26:54.477143 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.477144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.477158 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.477203 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.477778 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46054: BlockingRecv error: recv got EOF from 127.0.0.1:46054 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 02:26:54.609570 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.607127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.11.125.61:33713)
0925 02:26:54.607225 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.607227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.607281 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:26:54.607332 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.607334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.607344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.607394 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.607527 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.608006 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.608876 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.609108 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.609165 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.609308 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.609335 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.609373 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.609440 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.609456 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 02:26:54.611867 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.611192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.11.125.60:43507)
0925 02:26:54.611268 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.611271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.611297 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:26:54.611410 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.611411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.611423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.611474 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.611803 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46066: BlockingRecv error: recv got EOF from 127.0.0.1:46066 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:26:54.871555 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:54.869217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.11.125.61:33713)
0925 02:26:54.869289 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.869291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.869304 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:26:54.869363 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.869363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.869374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.869428 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:54.869563 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.870143 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.870891 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:54.871100 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:54.871156 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:54.871286 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:54.871315 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:54.871372 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:54.871417 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 02:26:54.871433 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:26:54.874013 1984 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:54.873403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.11.125.60:43507)
0925 02:26:54.873473 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:54.873475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:54.873485 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:54.873548 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:54.873549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:54.873559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:54.873602 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:54.873930 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46074: BlockingRecv error: recv got EOF from 127.0.0.1:46074 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:26:55.187947 1997 catalog_manager.cc:1259] Loaded cluster ID: b057da0aef224d3c9e9d11dce9f7aead
I20260925 02:26:55.188030 1997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: loading cluster ID for follower catalog manager: success
I20260925 02:26:55.188808 1997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: acquiring CA information for follower catalog manager: success
I20260925 02:26:55.189175 1997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 169564000f6f428ebc10b6f5974031d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:26:55.389596 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:55.387147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.11.125.61:33713)
0925 02:26:55.387265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:55.387267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:55.387289 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:26:55.387380 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:55.387381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:55.387393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:55.387458 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:55.387588 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.388088 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.388925 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.389116 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.389168 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:55.389300 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:55.389328 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:55.389373 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:55.389499 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 02:26:55.389519 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260925 02:26:55.394129 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:55.391390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.11.125.60:43507)
0925 02:26:55.391526 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:55.391528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:55.391541 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:26:55.391605 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:55.391606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:55.391688 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:55.391775 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:55.391898 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.392462 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.393343 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.393550 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.393614 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:55.393763 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:55.393879 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:55.393946 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:55.393981 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 02:26:55.394001 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260925 02:26:55.399644 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:55.396061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.11.125.62:39811)
0925 02:26:55.396231 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:55.396233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:55.396242 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:55.396295 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:55.396296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:55.396368 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:55.396649 (+ 281us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:55.396699 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.397559 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.398626 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.398913 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.398994 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:55.399219 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:55.399344 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:55.399465 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:55.399495 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:26:55.399524 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260925 02:26:55.399753 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:55.396139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.11.125.60:43507)
0925 02:26:55.396177 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:55.396179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:55.396435 (+ 256us) server_negotiation.cc:407] Connection header received
0925 02:26:55.396497 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:55.396498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:55.396508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:55.396571 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:55.396762 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.397566 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.398896 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.399198 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.399270 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:55.399431 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:55.399478 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:55.399524 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:55.399635 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:26:55.399657 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:55.399875 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:55.396230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.11.125.61:33713)
0925 02:26:55.396273 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:55.396275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:55.396438 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:26:55.396577 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:55.396578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:55.396685 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:55.396729 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:55.397042 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.397590 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.398866 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:55.399193 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:55.399268 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:55.399499 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:55.399533 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:55.399569 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:55.399766 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:26:55.399795 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:26:55.402017 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1759
I20260925 02:26:55.408550 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1835
I20260925 02:26:55.413815 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 1911
2026-09-25T02:26:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2867 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:26:55 dist-test-slave-hj7s krb5kdc[2181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:26:55 dist-test-slave-hj7s krb5kdc[2181](info): set up 2 sockets
Sep 25 02:26:55 dist-test-slave-hj7s krb5kdc[2181](info): commencing operation
krb5kdc: starting...
W20260925 02:26:56.471900 11764 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/test-user.keytab.
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.125.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
Entry for principal kudu/127.11.125.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab.
I20260925 02:26:56.592278 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43049
--webserver_interface=127.11.125.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:26:56.679558 2194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:56.679842 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:56.679878 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:56.679926 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:56.681213 2194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:56.681277 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:56.681304 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:56.681337 2194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:56.681366 2194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:56.682901 2194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.62.keytab
--principal=kudu/127.11.125.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.62:43049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.125.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:56.683190 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:56.683418 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:56.688531 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:56.688560 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:56.688627 2194 server_base.cc:1034] running on GCE node
W20260925 02:26:56.688572 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:56.689037 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:56.689349 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:56.690529 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790303216690516 us; error 41 us; skew 500 ppm
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:56.697914 2194 init.cc:377] Logged in from keytab as kudu/127.11.125.62@KRBTEST.COM (short username kudu)
I20260925 02:26:56.698391 2194 webserver.cc:460] Webserver started at http://127.11.125.62:38463/ using document root <none> and password file <none>
I20260925 02:26:56.698575 2194 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:56.698613 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:56.698709 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:56.699751 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data/instance:
uuid: "41f67b9f71e449e78ee8b79c620f162d"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "3992f7bf8f5a61d1f67f5330f6f3778f"
server_key_iv: "ecdc5a4c14943fb4178523bdd4bd1a64"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.700048 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance:
uuid: "41f67b9f71e449e78ee8b79c620f162d"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "3992f7bf8f5a61d1f67f5330f6f3778f"
server_key_iv: "ecdc5a4c14943fb4178523bdd4bd1a64"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.701349 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:56.702049 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:56.702222 2194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 02:26:56.702316 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal
uuid: "41f67b9f71e449e78ee8b79c620f162d"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "3992f7bf8f5a61d1f67f5330f6f3778f"
server_key_iv: "ecdc5a4c14943fb4178523bdd4bd1a64"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.702375 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:56.727475 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:56.729866 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:56.729981 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:56.734076 2194 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.62:43049
I20260925 02:26:56.734112 2261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.62:43049 every 8 connection(s)
I20260925 02:26:56.734464 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/data/info.pb
I20260925 02:26:56.735492 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:56.737162 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:56.737614 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 2194
I20260925 02:26:56.737737 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-0/wal/instance
I20260925 02:26:56.737975 11764 external_mini_cluster.cc:1428] Setting key 13b8dd95a5704bfbdc55791adcd95da5
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.61@KRBTEST.COM" created.
I20260925 02:26:56.748170 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.737955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.11.125.62:43049)
0925 02:26:56.738166 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:56.738177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:56.738334 (+ 157us) server_negotiation.cc:407] Connection header received
0925 02:26:56.738931 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:56.738933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:56.738964 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:56.739071 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:56.739651 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.740468 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.741529 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.741839 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.741978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.744116 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:56.744123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:56.744124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:56.744131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:56.746315 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.746760 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.746763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.746763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.746885 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.747150 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.747151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.747152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.747433 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:56.747549 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:56.747762 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:26:56.747913 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260925 02:26:56.748170 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.737939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43049 (local address 127.0.0.1:51248)
0925 02:26:56.738188 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.738225 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:56.738759 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:56.739075 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:56.739078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:56.739367 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:56.739520 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.739525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.740582 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.740583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.741423 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.741426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.741969 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.741970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.742036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.742584 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:56.742599 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:56.744008 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:56.746435 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.746442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.746444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.746657 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.746962 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.746963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.746963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.747059 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.747527 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:56.747533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:56.747602 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 02:26:56.747722 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:26:56.747984 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":113,"spinlock_wait_cycles":1280,"thread_start_us":105,"threads_started":1}
I20260925 02:26:56.749018 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:56.749408 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.749259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45798)
0925 02:26:56.749313 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.749375 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 02:26:56.749617 2210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
I20260925 02:26:56.750169 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.750012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45804)
0925 02:26:56.750100 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.750136 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:26:56.750357 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:34099: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
Entry for principal kudu/127.11.125.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab.
I20260925 02:26:56.763509 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:34099
--webserver_interface=127.11.125.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:56.799563 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } attempt: 1
I20260925 02:26:56.800158 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.799933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45806)
0925 02:26:56.800024 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.800101 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 02:26:56.800941 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.800783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45818)
0925 02:26:56.800846 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.800895 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:26:56.801038 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:34099: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
W20260925 02:26:56.845615 2269 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:56.845826 2269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:56.845844 2269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:56.845863 2269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:26:56.846922 2269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:56.846959 2269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:56.846974 2269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:56.846987 2269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:56.846998 2269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:56.848389 2269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.61.keytab
--principal=kudu/127.11.125.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.61:34099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.125.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:56.848588 2269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:56.848754 2269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:56.853703 2277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:56.853755 2275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:56.853868 2269 server_base.cc:1034] running on GCE node
W20260925 02:26:56.853750 2274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:56.854171 2269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:56.854461 2269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:56.855608 2269 hybrid_clock.cc:648] HybridClock initialized: now 1790303216855580 us; error 49 us; skew 500 ppm
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:56.858173 2269 init.cc:377] Logged in from keytab as kudu/127.11.125.61@KRBTEST.COM (short username kudu)
I20260925 02:26:56.858634 2269 webserver.cc:460] Webserver started at http://127.11.125.61:41507/ using document root <none> and password file <none>
I20260925 02:26:56.858816 2269 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:56.858856 2269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:56.858943 2269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:56.859943 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data/instance:
uuid: "bf5c755f5f614d33ae51e95ad18807f2"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "b62d382a9a24bd7403f8607f0cf45382"
server_key_iv: "a67232c0a9d6a4c2c30dc4e785f3c62a"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.860252 2269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal/instance:
uuid: "bf5c755f5f614d33ae51e95ad18807f2"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "b62d382a9a24bd7403f8607f0cf45382"
server_key_iv: "a67232c0a9d6a4c2c30dc4e785f3c62a"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.861696 2269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:56.862380 2284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:56.862561 2269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:56.862715 2269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal
uuid: "bf5c755f5f614d33ae51e95ad18807f2"
format_stamp: "Formatted at 2026-09-25 02:26:56 on dist-test-slave-hj7s"
server_key: "b62d382a9a24bd7403f8607f0cf45382"
server_key_iv: "a67232c0a9d6a4c2c30dc4e785f3c62a"
server_key_version: "encryptionkey@0"
I20260925 02:26:56.862782 2269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:56.869165 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } attempt: 2
I20260925 02:26:56.869562 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.869443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45820)
0925 02:26:56.869500 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.869536 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:26:56.869916 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.869818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45836)
0925 02:26:56.869865 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.869895 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 02:26:56.870003 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.61:34099: Network error: Client connection negotiation failed: client connection to 127.11.125.61:34099: connect: Connection refused (error 111)
I20260925 02:26:56.878794 2269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:56.880731 2269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:56.880889 2269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:56.884974 2269 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.61:34099
I20260925 02:26:56.885006 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.61:34099 every 8 connection(s)
I20260925 02:26:56.885355 2269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/data/info.pb
I20260925 02:26:56.886307 2338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:56.887847 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:56.888192 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 2269
I20260925 02:26:56.888301 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-1/wal/instance
I20260925 02:26:56.888494 11764 external_mini_cluster.cc:1428] Setting key 9c071200b00e975e29d24a5526de79a8
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.125.60@KRBTEST.COM; defaulting to no policy
I20260925 02:26:56.898285 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.888665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.11.125.62:43049)
0925 02:26:56.888739 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:56.888742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:56.888991 (+ 249us) server_negotiation.cc:407] Connection header received
0925 02:26:56.889496 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:56.889498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:56.889512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:56.889590 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:56.890446 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.890935 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.892474 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.892686 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.892741 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.895309 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:56.895311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:56.895311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:56.895320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:56.896952 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.897414 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.897416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.897417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.897483 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.897702 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.897703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.897703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.897808 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:56.897863 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:56.898079 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:26:56.898208 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:26:56.898486 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.888650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43049 (local address 127.0.0.1:51256)
0925 02:26:56.888836 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.888879 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:56.889402 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:56.889641 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:56.889645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:56.890029 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:56.890344 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.890351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.891027 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.891028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.892379 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.892383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.892787 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.892789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.892987 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.893481 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:56.893502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:56.895203 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:56.897046 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.897053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.897054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.897332 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.897554 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.897555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.897555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.897658 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.897862 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:56.897866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:56.897982 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 02:26:56.898060 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:26:56.898180 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260925 02:26:56.899060 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.125.60@KRBTEST.COM" created.
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:56.906585 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.899351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45838)
0925 02:26:56.899410 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.899423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:56.899495 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:56.899766 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:56.899767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:56.899911 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:56.899959 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.899960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.900811 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.900812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.901378 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.901380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.901872 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.901873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.901940 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.902369 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:56.902374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:56.903520 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:56.905450 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.905451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.905451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.905613 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.905850 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.905850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.905851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.905906 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.906233 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:56.906233 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:56.906271 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:26:56.906350 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 02:26:56.906450 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 02:26:56.906611 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.899430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.11.125.61:34099)
0925 02:26:56.899609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:56.899614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:56.899636 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:26:56.899657 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:56.899659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:56.899672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:56.899766 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:56.900042 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.900715 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.901461 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.901777 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.901873 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.903613 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:56.903622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:56.903622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:56.903629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:56.905327 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.905706 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.905709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.905710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.905767 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.905975 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.905976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.905977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.906157 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:56.906256 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:56.906382 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 02:26:56.906502 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260925 02:26:56.907318 2338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:56.907685 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.907552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58166)
0925 02:26:56.907611 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.907655 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:26:56.907821 2285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
I20260925 02:26:56.908826 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.908720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58180)
0925 02:26:56.908770 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.908797 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 02:26:56.908960 2338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:44285: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
Entry for principal kudu/127.11.125.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab.
I20260925 02:26:56.915383 11764 external_mini_cluster.cc:1326] Running /tmp/dist-test-task70c4Wq/build/release/bin/kudu
/tmp/dist-test-task70c4Wq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:44285
--webserver_interface=127.11.125.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.125.20:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:26:56.958184 2338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } attempt: 1
I20260925 02:26:56.958845 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.958690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58192)
0925 02:26:56.958761 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.958803 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:26:56.959263 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.959144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58206)
0925 02:26:56.959206 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.959236 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:26:56.959386 2338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:44285: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
I20260925 02:26:56.961130 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } attempt: 3
Sep 25 02:26:56 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:56.968226 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.961481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.11.125.61:34099)
0925 02:26:56.961526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:56.961528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:56.961539 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:56.961608 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:56.961609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:56.961622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:56.961672 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:56.962003 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.962510 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.963222 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.963470 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.963559 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.965683 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:56.965684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:56.965684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:56.965691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:56.967164 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.967539 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.967540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.967540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.967592 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:56.967784 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:56.967785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:56.967785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:56.967900 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:56.967951 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:56.968108 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:26:56.968167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 02:26:56.968287 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.961423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45842)
0925 02:26:56.961477 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.961487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:56.961541 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:56.961677 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:56.961678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:56.961852 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:56.961920 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.961921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.962607 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.962608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.963137 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:56.963139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:56.963561 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:56.963563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:56.963647 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:56.964111 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:56.964124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:56.965606 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:56.967245 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.967246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.967246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.967464 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.967657 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:56.967658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:56.967658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:56.967724 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:56.967964 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:56.967964 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:56.968016 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 02:26:56.968093 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:26:56.968190 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:26:56.968672 2262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:26:56.969059 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.968956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58212)
0925 02:26:56.968995 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.969032 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 02:26:56.969408 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:56.969282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58218)
0925 02:26:56.969360 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:56.969388 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 02:26:56.969521 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:44285: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
I20260925 02:26:57.000672 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } attempt: 1
W20260925 02:26:57.000783 2345 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:26:57.001097 2345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:26:57.001142 2345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:26:57.001194 2345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:26:57.001194 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.001036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58226)
0925 02:26:57.001109 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.001156 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 02:26:57.001555 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.001462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58234)
0925 02:26:57.001507 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.001535 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:26:57.001672 2262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:44285: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
W20260925 02:26:57.002552 2345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:26:57.002595 2345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:26:57.002610 2345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:26:57.002622 2345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:26:57.002674 2345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:26:57.004135 2345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.125.20:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.125.62:43049,127.11.125.61:34099,127.11.125.60:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/krb5kdc/kudu_127.11.125.60.keytab
--principal=kudu/127.11.125.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.125.60:44285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.125.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 01:40:19 UTC on ccycloud.quasar-gxqehx.root.comops.site
build id 4671765
I20260925 02:26:57.004418 2345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:26:57.004670 2345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:26:57.009632 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:26:57.009696 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:57.009819 2345 server_base.cc:1034] running on GCE node
W20260925 02:26:57.009707 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:26:57.010097 2345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:26:57.010386 2345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:26:57.011556 2345 hybrid_clock.cc:648] HybridClock initialized: now 1790303217011521 us; error 45 us; skew 500 ppm
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303217, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:26:57.014094 2345 init.cc:377] Logged in from keytab as kudu/127.11.125.60@KRBTEST.COM (short username kudu)
I20260925 02:26:57.014568 2345 webserver.cc:460] Webserver started at http://127.11.125.60:46763/ using document root <none> and password file <none>
I20260925 02:26:57.014797 2345 fs_manager.cc:359] Metadata directory not provided
I20260925 02:26:57.014874 2345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:26:57.015028 2345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:26:57.016119 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data/instance:
uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb"
format_stamp: "Formatted at 2026-09-25 02:26:57 on dist-test-slave-hj7s"
server_key: "1203618980f95a490868d849893270d5"
server_key_iv: "011a375ee55429c3d034b82885fffa2a"
server_key_version: "encryptionkey@0"
I20260925 02:26:57.016485 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal/instance:
uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb"
format_stamp: "Formatted at 2026-09-25 02:26:57 on dist-test-slave-hj7s"
server_key: "1203618980f95a490868d849893270d5"
server_key_iv: "011a375ee55429c3d034b82885fffa2a"
server_key_version: "encryptionkey@0"
I20260925 02:26:57.017903 2345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 02:26:57.018615 2361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:57.018783 2345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 02:26:57.018901 2345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data,/tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal
uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb"
format_stamp: "Formatted at 2026-09-25 02:26:57 on dist-test-slave-hj7s"
server_key: "1203618980f95a490868d849893270d5"
server_key_iv: "011a375ee55429c3d034b82885fffa2a"
server_key_version: "encryptionkey@0"
I20260925 02:26:57.018980 2345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:26:57.027570 2338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } attempt: 2
I20260925 02:26:57.028007 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.027879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58238)
0925 02:26:57.027942 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.027979 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:26:57.028369 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.028248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58246)
0925 02:26:57.028318 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.028350 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:26:57.028453 2338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.125.60:44285: Network error: Client connection negotiation failed: client connection to 127.11.125.60:44285: connect: Connection refused (error 111)
I20260925 02:26:57.034263 2345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:26:57.036334 2345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:26:57.036502 2345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:26:57.040804 2345 rpc_server.cc:307] RPC server started. Bound to: 127.11.125.60:44285
I20260925 02:26:57.040845 2413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.125.60:44285 every 8 connection(s)
I20260925 02:26:57.041251 2345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/data/info.pb
I20260925 02:26:57.042222 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:26:57.043726 2414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303217, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
I20260925 02:26:57.050483 11764 external_mini_cluster.cc:1388] Started /tmp/dist-test-task70c4Wq/build/release/bin/kudu as pid 2345
I20260925 02:26:57.050616 11764 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task70c4Wq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790302942477976-11764-0/minicluster-data/master-2/wal/instance
I20260925 02:26:57.050828 11764 external_mini_cluster.cc:1428] Setting key 38294ba3aad370632242f263a3185aff
I20260925 02:26:57.052680 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.044427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.11.125.62:43049)
0925 02:26:57.044498 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.044500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.044675 (+ 175us) server_negotiation.cc:407] Connection header received
0925 02:26:57.045209 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.045211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.045220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.045280 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.045894 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.046363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.047304 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.047481 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.047531 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.049608 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.049610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.049611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.049618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.051191 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.051640 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.051642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.051642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.051710 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.052013 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.052014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.052015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.052139 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.052209 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.052502 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.052590 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:26:57.052866 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.044378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.62:43049 (local address 127.0.0.1:51272)
0925 02:26:57.044545 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.044586 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:57.045117 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:57.045352 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:57.045355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:57.045607 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:57.045800 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.045804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.046447 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.046447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.047226 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.047228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.047563 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.047564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.047673 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.048118 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:57.048126 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:57.049519 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:57.051287 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.051291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.051293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.051554 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.051784 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.051785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.051786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.051932 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.052214 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:57.052221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:57.052361 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 02:26:57.052464 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:26:57.052622 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260925 02:26:57.053664 2414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303217, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:57.060402 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.051261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.11.125.60:44285)
0925 02:26:57.051470 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.051475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.051496 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:26:57.051524 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.051527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.051541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.051608 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.052362 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.053276 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.054208 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.054522 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.054604 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.057217 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.057225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.057225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.057231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.059043 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.059404 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.059407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.059408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.059472 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.059729 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.059729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.059730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.059938 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.060092 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.060174 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.060267 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260925 02:26:57.062062 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.054058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.11.125.61:34099)
0925 02:26:57.054129 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.054132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.054145 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:26:57.054176 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.054177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.054188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.054267 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.054607 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.055153 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.055982 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.056176 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.056234 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.058649 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.058651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.058651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.058659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.060849 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.061283 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.061284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.061284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.061358 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.061555 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.061556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.061557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.061675 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.061751 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.061900 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.061975 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:26:57.062057 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.053979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.61:34099 (local address 127.0.0.1:45852)
0925 02:26:57.054034 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.054046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:57.054096 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:57.054284 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:57.054285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:57.054448 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:57.054501 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.054503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.055264 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.055265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.055829 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.055831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.056260 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.056261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.056359 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.057248 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:57.057253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:57.058544 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:57.060977 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.060978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.060978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.061188 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.061432 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.061433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.061433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.061489 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.061744 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:57.061745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:57.061786 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:26:57.061872 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:26:57.061962 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:26:57.062716 2414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.62@KRBTEST.COM
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303217, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.60@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:57.068426 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.061198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.11.125.62:43049)
0925 02:26:57.061249 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.061251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.061261 (+ 10us) server_negotiation.cc:407] Connection header received
0925 02:26:57.061284 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.061284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.061295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.061358 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.061777 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.062310 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.063142 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.063348 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.063420 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.065565 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.065567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.065567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.065575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.067281 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.067652 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.067653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.067653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.067714 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.067908 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.067909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.067910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.068054 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.068241 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.068275 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.068343 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 02:26:57.069255 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.063009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58262)
0925 02:26:57.063104 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.063113 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:57.063160 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:57.063279 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:57.063279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:57.063409 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:57.063433 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.063434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.064038 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.064039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.064488 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.064489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.064876 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.064877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.064950 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.065341 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:57.065346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:57.066484 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:57.068148 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.068149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.068149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.068365 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.068568 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.068569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.068569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.068625 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.068878 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:57.068878 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:57.068953 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 02:26:57.069027 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:26:57.069120 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 02:26:57.069370 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.063069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.11.125.60:44285)
0925 02:26:57.063102 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.063103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.063146 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:26:57.063212 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.063213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.063223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.063275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.063503 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.063951 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.064563 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.064788 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.064875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.066551 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.066553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.066553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.066560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.068071 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.068442 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.068443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.068443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.068501 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.068679 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.068681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.068681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.068820 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.068878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.069075 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.069186 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":15}
I20260925 02:26:57.070894 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: Bootstrap starting.
I20260925 02:26:57.071955 2414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:57.072302 2414 log.cc:824] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:57.073208 2414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: No bootstrap required, opened a new log
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.125.61@KRBTEST.COM
I20260925 02:26:57.075230 2414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.075426 2414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f0a0aa5eb24c3a9b111f4274fd1fdb, State: Initialized, Role: FOLLOWER
I20260925 02:26:57.075519 2414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.075914 2414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:57.076794 2262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } attempt: 2
I20260925 02:26:57.077518 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.077623 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.077730 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.069172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.11.125.61:34099)
0925 02:26:57.069210 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.069211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.069235 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:26:57.069256 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.069256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.069267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.069321 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.070186 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.070641 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.071664 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.071983 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.072081 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.074513 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.074514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.074515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.074523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.076210 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.076566 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.076568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.076568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.076637 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.076953 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.076954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.076954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.077104 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.077269 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.077529 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.077607 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 02:26:57.079286 2432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:26:57.079391 2432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:57.080030 11764 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:26:57.078620 2345 master_runner.cc:186] Error getting master registration for 127.11.125.62:43049: OK, instance_id { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" instance_seqno: 1790303216732858 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:26:57.080967 2345 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.62@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:57.081374 2236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41f67b9f71e449e78ee8b79c620f162d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51274
I20260925 02:26:57.085115 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.077146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.11.125.60:44285)
0925 02:26:57.077305 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.077307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.077316 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:57.077348 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.077349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.077358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.077402 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.078031 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.078627 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.079544 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.079826 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.079900 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.082209 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.082211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.082211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.082218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.083831 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.084227 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.084228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.084228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.084290 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.084509 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.084510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.084510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.084680 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.084757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.084945 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.085028 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 02:26:57.085141 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.077086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58276)
0925 02:26:57.077187 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.077199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:57.077238 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:57.077422 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:57.077423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:57.077591 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:57.077638 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.077640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.078716 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.078717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.079458 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.079461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.079956 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.079957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.080021 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.080478 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:57.080484 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:57.082109 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:57.083920 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.083921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.083922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.084146 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.084373 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.084373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.084374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.084441 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.084755 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:57.084755 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:57.084802 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:26:57.084871 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:26:57.085036 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 02:26:57.086797 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d: Bootstrap starting.
I20260925 02:26:57.087570 2262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:57.087922 2262 log.cc:824] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:57.088784 2262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d: No bootstrap required, opened a new log
I20260925 02:26:57.090169 2262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.090317 2262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f67b9f71e449e78ee8b79c620f162d, State: Initialized, Role: FOLLOWER
I20260925 02:26:57.090411 2262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.090708 2262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:57.090665 2435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.090755 2236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:26:57.090777 2435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.090814 2236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:57.091540 2236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.091754 2236 leader_election.cc:288] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [CANDIDATE]: Term 1 election: Requested vote from peers bf5c755f5f614d33ae51e95ad18807f2 (127.11.125.61:34099), 60f0a0aa5eb24c3a9b111f4274fd1fdb (127.11.125.60:44285)
I20260925 02:26:57.092298 2388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f67b9f71e449e78ee8b79c620f162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb"
I20260925 02:26:57.092413 2388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:57.093449 2388 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f67b9f71e449e78ee8b79c620f162d in term 1.
I20260925 02:26:57.093674 2210 leader_election.cc:302] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f67b9f71e449e78ee8b79c620f162d, 60f0a0aa5eb24c3a9b111f4274fd1fdb; no voters:
I20260925 02:26:57.093844 2435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:26:57.094043 2435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [term 1 LEADER]: Becoming Leader. State: Replica: 41f67b9f71e449e78ee8b79c620f162d, State: Running, Role: LEADER
I20260925 02:26:57.094146 2435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.095433 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f67b9f71e449e78ee8b79c620f162d. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.095525 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:57.095748 2456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:26:57.096586 2456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:26:57.093989 2194 master_runner.cc:186] Error getting master registration for 127.11.125.61:34099: OK, instance_id { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" instance_seqno: 1790303216883819 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:26:57.096879 2194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:26:57.102319 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.093075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.11.125.62:43049)
0925 02:26:57.093201 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.093203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.093429 (+ 226us) server_negotiation.cc:407] Connection header received
0925 02:26:57.094041 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.094042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.094054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.094085 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.095054 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.095777 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.097097 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.097402 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.097444 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.099142 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.099144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.099144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.099151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.100720 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.101375 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.101377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.101377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.101447 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.101672 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.101673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.101673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.101815 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.101886 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.102132 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.102221 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 02:26:57.104647 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.094696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.11.125.61:34099)
0925 02:26:57.094742 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.094744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.095741 (+ 997us) server_negotiation.cc:407] Connection header received
0925 02:26:57.095855 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.095856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.095868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.095897 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.096598 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.097328 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.098294 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.098601 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.098677 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.101666 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.101667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.101667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.101675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.103277 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.103695 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.103696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.103697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.103758 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.104005 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.104006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.104007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.104161 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.104272 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.104423 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.104552 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:57.108323 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.095054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.11.125.60:44285)
0925 02:26:57.095090 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.095092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.098038 (+ 2946us) server_negotiation.cc:407] Connection header received
0925 02:26:57.099186 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.099187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.099199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.099266 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.099849 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.100551 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.103546 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.103856 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.103922 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.105559 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.105561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.105561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.105568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.107122 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.107533 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.107534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.107534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.107592 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.107792 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.107793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.107794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.107927 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.108014 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.108158 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.108234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 02:26:57.111821 2388 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [term 1 FOLLOWER]: Refusing update from remote peer 41f67b9f71e449e78ee8b79c620f162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:57.112146 2435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:57.113924 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f67b9f71e449e78ee8b79c620f162d. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.114017 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.117940 2458 mvcc.cc:205] Tried to move back new op lower bound from 7333081977279463424 to 7333081977218469888. Current Snapshot: MvccSnapshot[applied={T|T < 7333081977279463424}]
I20260925 02:26:57.118005 2435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.118088 2435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:57.118752 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.118829 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.118978 2459 mvcc.cc:205] Tried to move back new op lower bound from 7333081977279463424 to 7333081977218469888. Current Snapshot: MvccSnapshot[applied={T|T < 7333081977279463424 or (T in {7333081977279463424})}]
I20260925 02:26:57.119593 2338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } attempt: 3
I20260925 02:26:57.122989 2456 catalog_manager.cc:1347] Generated new cluster ID: 827712aff43446759e91a9c6170ecd63
I20260925 02:26:57.123040 2456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 02:26:57 dist-test-slave-hj7s krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790303216, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.125.61@KRBTEST.COM for kudu/127.11.125.60@KRBTEST.COM
I20260925 02:26:57.127835 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.119832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.125.60:44285 (local address 127.0.0.1:58290)
0925 02:26:57.119888 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 02:26:57.119902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:26:57.119951 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:26:57.120088 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:26:57.120088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:26:57.120267 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:26:57.120317 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.120319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.121330 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.121331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.122169 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:26:57.122172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.123050 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.123051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:26:57.123116 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.123558 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:26:57.123563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:26:57.125060 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:26:57.126776 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.126777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.126777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.126996 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.127201 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:26:57.127202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:26:57.127202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:26:57.127257 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:26:57.127522 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:26:57.127523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:26:57.127566 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:26:57.127627 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 02:26:57.127723 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:26:57.128194 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:57.119918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.11.125.60:44285)
0925 02:26:57.119955 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.119957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.119966 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:57.120006 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.120007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.120017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.120075 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.120409 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.121226 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.122265 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:57.122602 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:57.122659 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:57.125154 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:26:57.125155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:26:57.125156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:26:57.125163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:26:57.126683 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.127076 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.127077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.127077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.127133 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:26:57.127318 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:26:57.127319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:26:57.127319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:26:57.127442 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:26:57.127498 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:57.127997 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 02:26:57.128070 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 02:26:57.129739 2338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: Bootstrap starting.
I20260925 02:26:57.130509 2338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: Neither blocks nor log segments found. Creating new log.
I20260925 02:26:57.130824 2338 log.cc:824] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:26:57.137633 2338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: No bootstrap required, opened a new log
I20260925 02:26:57.139716 2338 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.139866 2338 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5c755f5f614d33ae51e95ad18807f2, State: Initialized, Role: FOLLOWER
I20260925 02:26:57.139961 2338 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } }
I20260925 02:26:57.140331 2338 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:26:57.140416 2310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:26:57.141448 2310 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [term 1 FOLLOWER]: Refusing update from remote peer 41f67b9f71e449e78ee8b79c620f162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:26:57.141629 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f67b9f71e449e78ee8b79c620f162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf5c755f5f614d33ae51e95ad18807f2"
I20260925 02:26:57.142278 2452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:26:57.142376 2312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f67b9f71e449e78ee8b79c620f162d in term 1.
I20260925 02:26:57.144721 2476 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.144829 2476 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.144888 2476 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f67b9f71e449e78ee8b79c620f162d. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.144968 2476 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.145004 2476 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.145048 2476 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:26:57.149835 2475 catalog_manager.cc:1259] Loaded cluster ID: 827712aff43446759e91a9c6170ecd63
I20260925 02:26:57.149900 2475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: loading cluster ID for follower catalog manager: success
W20260925 02:26:57.150002 2475 catalog_manager.cc:1576] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 02:26:57.150053 2475 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:26:57.159926 2452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f67b9f71e449e78ee8b79c620f162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f67b9f71e449e78ee8b79c620f162d" member_type: VOTER last_known_addr { host: "127.11.125.62" port: 43049 } } peers { permanent_uuid: "bf5c755f5f614d33ae51e95ad18807f2" member_type: VOTER last_known_addr { host: "127.11.125.61" port: 34099 } } peers { permanent_uuid: "60f0a0aa5eb24c3a9b111f4274fd1fdb" member_type: VOTER last_known_addr { host: "127.11.125.60" port: 44285 } } }
I20260925 02:26:57.160044 2452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d [sys.catalog]: This master's current role is: LEADER
I20260925 02:26:57.162911 2456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:26:57.163686 2456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:26:57.167670 2456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f67b9f71e449e78ee8b79c620f162d: Generated new TSK 0
I20260925 02:26:57.167867 2456 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:26:57.194890 2265 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.193225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.11.125.62:43049)
0925 02:26:57.193269 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.193272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.194205 (+ 933us) server_negotiation.cc:407] Connection header received
0925 02:26:57.194315 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.194317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.194327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.194359 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.194808 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51302: BlockingRecv error: recv got EOF from 127.0.0.1:51302 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:26:57.195819 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.193628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.11.125.61:34099)
0925 02:26:57.193679 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.193681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.195296 (+ 1615us) server_negotiation.cc:407] Connection header received
0925 02:26:57.195405 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.195407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.195418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.195452 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.195757 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45880: BlockingRecv error: recv got EOF from 127.0.0.1:45880 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 02:26:57.196621 2418 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.193943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.11.125.60:44285)
0925 02:26:57.193982 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.193984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.196116 (+ 2132us) server_negotiation.cc:407] Connection header received
0925 02:26:57.196228 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.196229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.196241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.196271 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.196563 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58304: BlockingRecv error: recv got EOF from 127.0.0.1:58304 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:26:57.198994 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.198265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.11.125.61:34099)
0925 02:26:57.198301 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.198303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.198473 (+ 170us) server_negotiation.cc:407] Connection header received
0925 02:26:57.198576 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.198577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.198595 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.198624 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.198930 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45882: BlockingRecv error: recv got EOF from 127.0.0.1:45882 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 02:26:57.202042 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.201343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.11.125.61:34099)
0925 02:26:57.201375 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.201377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.201580 (+ 203us) server_negotiation.cc:407] Connection header received
0925 02:26:57.201673 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.201673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.201684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.201712 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.201999 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45890: BlockingRecv error: recv got EOF from 127.0.0.1:45890 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 02:26:57.206055 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.205315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.11.125.61:34099)
0925 02:26:57.205348 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.205350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.205592 (+ 242us) server_negotiation.cc:407] Connection header received
0925 02:26:57.205683 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.205683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.205693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.205720 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.205997 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45892: BlockingRecv error: recv got EOF from 127.0.0.1:45892 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 02:26:57.212080 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.211300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.11.125.61:34099)
0925 02:26:57.211338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.211353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.211569 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:26:57.211666 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.211667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.211678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.211705 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.212035 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45902: BlockingRecv error: recv got EOF from 127.0.0.1:45902 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:26:57.222615 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.221589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.11.125.61:34099)
0925 02:26:57.221633 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.221635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.221955 (+ 320us) server_negotiation.cc:407] Connection header received
0925 02:26:57.222063 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.222064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.222076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.222107 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.222561 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45910: BlockingRecv error: recv got EOF from 127.0.0.1:45910 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 02:26:57.241165 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.240177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.11.125.61:34099)
0925 02:26:57.240222 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.240224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.240499 (+ 275us) server_negotiation.cc:407] Connection header received
0925 02:26:57.240609 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.240610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.240624 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.240656 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.241094 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45922: BlockingRecv error: recv got EOF from 127.0.0.1:45922 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 02:26:57.275645 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.274591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.11.125.61:34099)
0925 02:26:57.274639 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.274641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.274888 (+ 247us) server_negotiation.cc:407] Connection header received
0925 02:26:57.275004 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.275006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.275022 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.275110 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.275578 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45926: BlockingRecv error: recv got EOF from 127.0.0.1:45926 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 02:26:57.341843 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.340938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.11.125.61:34099)
0925 02:26:57.341002 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.341004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.341105 (+ 101us) server_negotiation.cc:407] Connection header received
0925 02:26:57.341156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.341157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.341168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.341269 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.341774 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45938: BlockingRecv error: recv got EOF from 127.0.0.1:45938 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:26:57.471783 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.470985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.11.125.61:34099)
0925 02:26:57.471080 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.471082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.471178 (+ 96us) server_negotiation.cc:407] Connection header received
0925 02:26:57.471279 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.471281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.471292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.471354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.471718 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45946: BlockingRecv error: recv got EOF from 127.0.0.1:45946 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:26:57.729894 2342 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:26:57.729219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.11.125.61:34099)
0925 02:26:57.729292 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:57.729294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:57.729331 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:26:57.729400 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:57.729401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:57.729412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:57.729463 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:26:57.729823 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45958: BlockingRecv error: recv got EOF from 127.0.0.1:45958 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:26:58.079914 2432 catalog_manager.cc:1259] Loaded cluster ID: 827712aff43446759e91a9c6170ecd63
I20260925 02:26:58.079998 2432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: loading cluster ID for follower catalog manager: success
I20260925 02:26:58.080870 2432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: acquiring CA information for follower catalog manager: success
I20260925 02:26:58.081239 2432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60f0a0aa5eb24c3a9b111f4274fd1fdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:26:58.151260 2475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: acquiring CA information for follower catalog manager: success
I20260925 02:26:58.151723 2475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf5c755f5f614d33ae51e95ad18807f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:26:58.246115 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:58.243154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.11.125.61:34099)
0925 02:26:58.243286 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:58.243288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:58.243301 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:26:58.243429 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:58.243431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:58.243492 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:58.243581 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:58.243811 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.244415 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.245195 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.245412 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.245528 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:58.245737 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:58.245863 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:58.245955 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:58.245980 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 02:26:58.246003 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 02:26:58.250475 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:58.247917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.11.125.60:44285)
0925 02:26:58.248027 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:58.248030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:58.248039 (+ 9us) server_negotiation.cc:407] Connection header received
0925 02:26:58.248120 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:58.248126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:58.248165 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:58.248266 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:58.248362 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.248840 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.249650 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.249859 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.249945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:58.250114 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:58.250226 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:58.250317 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:58.250347 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 02:26:58.250369 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260925 02:26:58.255012 2562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:58.252230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.11.125.62:43049)
0925 02:26:58.252357 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:58.252359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:58.252389 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:26:58.252438 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:58.252439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:58.252493 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:58.252569 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:58.252757 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.253352 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.254088 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.254356 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.254420 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:58.254682 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:58.254787 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:58.254844 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:58.254890 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 02:26:58.254913 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260925 02:26:58.255596 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:58.252513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.11.125.60:44285)
0925 02:26:58.252565 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:58.252567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:58.252578 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:26:58.252600 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:58.252601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:58.252612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:58.252671 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:58.252838 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.253581 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.254675 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.254961 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.255021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:58.255277 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:58.255311 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:58.255352 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:58.255502 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 02:26:58.255531 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:26:58.255615 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:26:58.252179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.11.125.61:34099)
0925 02:26:58.252223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 02:26:58.252225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 02:26:58.252442 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:26:58.252716 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:26:58.252718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:26:58.252729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:26:58.252759 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:26:58.253039 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.253812 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.254803 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:26:58.255103 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:26:58.255167 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:26:58.255354 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:26:58.255390 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:26:58.255416 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 02:26:58.255528 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:26:58.255549 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 02:26:58.257754 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 2194
I20260925 02:26:58.263461 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 2269
I20260925 02:26:58.268702 11764 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task70c4Wq/build/release/bin/kudu with pid 2345
2026-09-25T02:26:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2854 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5722 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275796 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST